Method and system for retrieving data in response to user input

ABSTRACT

A method of retrieving data comprises receiving a user input in the form of a command, determining that the user input is a gesture, sending a request to a cloud server based on the determined gesture, receiving data from a knowledge base connected to the cloud server based on the request, and displaying virtual content at the user device based on the received data.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of pending U.S. patent applicationSer. No. 14/696,347, entitled “PLANAR WAVEGUIDE APPARATUS WITHDIFFRACTION ELEMENT(S) AND SYSTEM EMPLOYING SAME”, filed Apr. 24, 2015,which is a continuation of U.S. patent application Ser. No. 14/331,218,entitled “PLANAR WAVEGUIDE APPARATUS WITH DIFFRACTION ELEMENT(S) ANDSYSTEM EMPLOYING SAME”, filed Jul. 14, 2014, which claims priority toU.S. Provisional Application Ser. No. 61/845,907, entitled “PLANARWAVEGUIDE APPARATUS WITH DIFFRACTION ELEMENT(S) AND SYSTEM EMPLOYINGSAME”, filed Jul. 12, 2013, and also claims priority to U.S. ProvisionalApplication Ser. No. 62/012,273, entitled “METHODS AND SYSTEMS FORCREATING VIRTUAL AND AUGMENTED REALITY”, filed on Jun. 14, 2014. Thisapplication is also a continuation-in-part of U.S. patent applicationSer. No. 14/641,376, entitled “VIRTUAL AND AUGMENTED REALITY SYSTEMS ANDMETHODS”, filed Mar. 7, 2015, which claims priority to U.S. ProvisionalApplication Ser. No. 61/950,001 filed Mar. 7, 2014. This application iscross-related to U.S. patent application Ser. No. 14/690,401, entitled“SYSTEMS AND METHOD FOR AUGMENTED REALITY”, filed Apr. 18, 2015 and toU.S. patent application Ser. No. 14/641,376, entitled “VIRTUAL ANDAUGMENTED REALITY SYSTEMS AND METHODS,” filed Mar. 7, 2015, and U.S.patent application Ser. No. 13/915,530, entitled “MULTIPLE DEPTH PLANETHREE-DIMENSIONAL DISPLAY USING A WAVE GUIDE REFLECTOR ARRAY PROJECTOR”,filed Jun. 11, 2013. This application is also cross-related to U.S.patent application Ser. No. 14/205,126, entitled “SYSTEM AND METHOD FORAUGMENTED AND VIRTUAL REALITY”, filed Mar. 11, 2014. The contents of theaforementioned patent applications are hereby expressly incorporated byreference in their entireties.

FIELD OF THE INVENTION

The present invention generally relates to systems and methodsconfigured to facilitate interactive virtual or augmented realityenvironments for one or more users.

BACKGROUND

A light field encompasses all the light rays at every point in spacetraveling in every direction. Light fields are considered fourdimensional because every point in a three-dimensional space also has anassociated direction, which is the fourth dimension.

Wearable three-dimensional displays may include a substrate guidedoptical device, also known as the light-guide optical element (LOE)system. Such devices are manufactured by, for example Lumus Ltd.However, these LOE systems only project a single depth plane, focused atinfinity, with a spherical wave front curvature of zero.

One prior art system (Lumus) comprises multiple angle-dependentreflectors embedded in a waveguide to outcouple light from the face ofthe waveguide. Another prior art system (BAE) embeds a lineardiffraction grating within the waveguide to change the angle of incidentlight propagating along the waveguide. By changing the angle of lightbeyond the threshold of TIR, the light escapes from one or more lateralfaces of the waveguide. The linear diffraction grating has a lowdiffraction efficiency, so only a fraction of the light energy isdirected out of the waveguide, each time the light encounters the lineardiffraction grating. By outcoupling the light at multiple locationsalong the grating, the exit pupil of the display system is effectivelyincreased.

A primary limitation of the prior art systems is that they only relaycollimated images to the eyes (i.e., images at optical infinity).Collimated displays are adequate for many applications in avionics,where pilots are frequently focused upon very distant objects (e.g.,distant terrain or other aircraft). However, for many other head-up oraugmented reality applications, it is desirable to allow users to focustheir eyes upon (i.e., “accommodate” to) objects closer than opticalinfinity.

The wearable 3D displays may be used for so called “virtual reality” or“augmented reality” experiences, wherein digitally reproduced images orportions thereof are presented to a user in a manner wherein they seemto be, or may be perceived as, real. A virtual reality, or “VR”,scenario typically involves presentation of digital or virtual imageinformation without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user.

The U.S. patent applications listed above present systems and techniquesto work with the visual configuration of a typical human to addressvarious challenges in virtual reality and augmented realityapplications. The design of these virtual reality and/or augmentedreality systems (AR systems) presents numerous challenges, including thespeed of the system in delivering virtual content, quality of virtualcontent, eye relief of the user, size and portability of the system, andother system and optical challenges.

The systems and techniques described herein are configured to work withthe visual configuration of the typical human to address thesechallenges.

SUMMARY

Embodiments of the present invention are directed to devices, systemsand methods for facilitating virtual reality and/or augmented realityinteraction for one or more users.

Light that is coupled into a planar waveguide (e.g., pane of glass, paneof fused silica, pane of polycarbonate), will propagate along thewaveguide by total internal reflection (TIR). Planar waveguides may alsobe referred to as “substrate-guided optical elements,” or “lightguides.”

If that light encounters one or more diffraction optical elements (DOE)in or adjacent to the planar waveguide, the characteristics of thatlight (e.g., angle of incidence, wavefront shape, wavelength, etc.) canbe altered such that a portion of the light escapes TIR and emerges fromone or more faces of the waveguide.

If the light coupled into the planar waveguide is varied spatiallyand/or temporally to contain or encode image data that image data canpropagate along the planar waveguide by TIR. Examples of elements thatspatially vary light include LCDs, LCoS panels, OLEDs, DLPs, and otherimage arrays. Typically, these spatial light modulators may update imagedata for different cells or sub-elements at different points in time,and thus may produce sub-frame temporal variation, in addition tochanging image data on a frame-by-frame basis to produce moving video.Examples of elements that temporally vary light include acousto-opticalmodulators, interferometric modulators, optical choppers, and directlymodulated emissive light sources such as LEDs and laser diodes. Thesetemporally varying elements may be coupled to one or more elements tovary the light spatially, such as scanning optical fibers, scanningmirrors, scanning prisms, and scanning cantilevers with reflectiveelements—or these temporally varying elements may be actuated directlyto move them through space. Such scanning systems may utilize one ormore scanned beams of light that are modulated over time and scannedacross space to display image data.

If image data contained in spatially and/or temporally varying lightthat propagates along a planar waveguide by TIR encounters one or moreDOEs in or adjacent to the planar waveguide, the characteristics of thatlight can be altered such that the image data encoded in light willescape TIR and emerge from one or more faces of the planar waveguide.Inclusion of one or more DOEs which combine a linear diffraction gratingfunction or phase pattern with a radially symmetric or circular lensfunction or phase pattern, may advantageously allow steering of beamsemanating from the face of the planar waveguide and control over focusor focal depth.

By incorporating such a planar waveguide system into a display system,the waveguide apparatus (e.g., planar waveguide and associated DOE) canbe used to present images to one or more eyes. Where the planarwaveguide is constructed of a partially or wholly transparent material,a human may view real physical objects through the waveguide. Thewaveguide display system can, thus, comprise an optically see-throughmixed reality (or “augmented reality”) display system, in whichartificial or remote image data can be superimposed, overlaid, orjuxtaposed with real scenes.

The structures and approaches described herein may advantageouslyproduce a relatively large eye box, readily accommodating viewer's eyemovements.

In another aspect, a method of rendering virtual content to a user isdisclosed. The method comprises detecting a location of a user,retrieving a set of data associated with a part of a virtual world modelthat corresponds to the detected location of the user, wherein thevirtual world model comprises data associated with a set of map pointsof the real world, and rendering, based on the set of retrieved data,virtual content to a user device of the user, such that the virtualcontent, when viewed by the user, appears to be placed in relation to aset of physical objects in a physical environment of the user.

In another aspect, a method of recognizing objects is disclosed. Themethod comprises capturing an image of a field of view of a user,extracting a set of map points based on the captured image, recognizingan object based on the extracted set of map points, retrieving semanticdata associated with the recognized objects and attaching the semanticdata to data associated with the recognized object and inserting therecognized object data attached with the semantic data to a virtualworld model such that virtual content is placed in relation to therecognized object.

In another aspect, a method comprises capturing an image of a field ofview of a user, extracting a set of map points based on the capturedimage, identifying a set of sparse points and dense points based on theextraction, performing point normalization on the set of sparse pointsand dense points, generating point descriptors for the set of sparsepoints and dense points, and combining the sparse point descriptors anddense point descriptors to store as map data.

In another aspect, a method of determining user input is disclosed. Inone embodiment, the method comprises capturing an image of a field ofview of a user, the image comprising a gesture created by the user,analyzing the captured image to identify a set of points associated withthe gesture, comparing the set of identified points to a set of pointsassociated with a database of predetermined gestures, generating ascoring value for the set of identified points based on the comparison,recognizing the gesture when the scoring value exceeds a thresholdvalue, and determining a user input based on the recognized gesture.

In another aspect, a method of determining user input is disclosed. Themethod comprises detecting a movement of a totem in relation to areference frame, recognizing a pattern based on the detected movement,comparing the recognizing pattern to a set of predetermined patterns,generating a scoring value for the recognized pattern based on thecomparison, recognizing the movement of the totem when the scoring valueexceeds a threshold value, and determining a user input based on therecognized movement of the totem.

In another aspect, a method of generating a virtual user interface isdisclosed. The method comprises identifying a virtual user interface tobe displayed to a user, generating a set of data associated with thevirtual user interface, tethering the virtual user interface to a set ofmap points associated with at least one physical entity at the user'slocation, and displaying the virtual user interface to the user, suchthat the virtual user interface, when viewed by the user, moves inrelation to a movement of the at least one physical entity.

In another aspect, a method comprises detecting a movement of a user'sfingers or a totem, recognizing, based on the detected movement, acommand to create a virtual user interface, determining, from a virtualworld model, a set of map points associated with a position of theuser's fingers or the totem, and rendering, in real-time, a virtual userinterface at the determined map points associated with the position ofthe user's fingers or the totem such that the user views the virtualuser interface being created simultaneously as the user's fingers ortotem move to define a location or outline of the virtual userinterface.

In another aspect, a method comprises identifying a real-world activityof a user; retrieving a knowledge base associated with the real-worldactivity, creating a virtual user interface in a field of view of theuser, and displaying, on the virtual user interface, a set ofinformation associated with the real-world activity based on theretrieved knowledge base.

In yet another aspect, a method comprises uploading a set of dataassociated with a physical environment of a first user to a virtualworld model residing in a cloud server, updating the virtual world modelbased on the uploaded data, transmitting a piece of the virtual worldmodel associated with the physical environment of the first user to asecond user located at a different location than the first user, anddisplaying, at a user device of the second user, a virtual copy of thephysical environment of the first user based on the transmitted piece ofthe virtual world model.

Additional and other objects, features, and advantages of the inventionare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing an optical system including awaveguide apparatus, a subsystem to couple light to or from thewaveguide apparatus, and a control subsystem, according to oneillustrated embodiment.

FIG. 2 an elevational view showing a waveguide apparatus including aplanar waveguide and at least one diffractive optical element positionedwithin the planar waveguide, illustrating a number of optical pathsincluding totally internally reflective optical paths and optical pathsbetween an exterior and an interior of the planar waveguide, accordingto one illustrated embodiment.

FIG. 3A a schematic diagram showing a linear diffraction or diffractivephase function, according to one illustrated embodiment.

FIG. 3B a schematic diagram showing a radially circular lens phasefunction, according to one illustrated embodiment.

FIG. 3C a schematic diagram showing a linear diffraction or diffractivephase function of a diffractive optical element that combines the lineardiffraction and the radially circular lens phase functions, thediffractive optical element associated with a planar waveguide.

FIG. 4A an elevational view showing a waveguide apparatus including aplanar waveguide and at least one diffractive optical element carried onan outer surface of the planar waveguide, according to one illustratedembodiment.

FIG. 4B an elevational view showing a waveguide apparatus including aplanar waveguide and at least one diffractive optical element positionedinternally immediately adjacent an outer surface of the planarwaveguide, according to one illustrated embodiment.

FIG. 4C an elevational view showing a waveguide apparatus including aplanar waveguide and at least one diffractive optical element formed inan outer surface of the planar waveguide, according to one illustratedembodiment.

FIG. 5A is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem,according to one illustrated embodiment.

FIG. 5B is a schematic diagram of the optical system of FIG. 5Aillustrating generation of a single focus plane that is capable of beingpositioned closer than optical infinity, according to one illustratedembodiment.

FIG. 5C is a schematic diagram of the optical system of FIG. 5Aillustrating generation of a multi-focal volumetric display, image orlight field, according to one illustrated embodiment.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem including a pluralityof projectors to optically couple light to a primary planar waveguide,according to one illustrated embodiment.

FIG. 7 is an elevational view of a planar waveguide apparatus includinga planar waveguide with a plurality of DOEs, according to oneillustrated embodiment.

FIG. 8 is an elevational view showing a portion of an optical systemincluding a plurality of planar waveguide apparati in a stacked array,configuration or arrangement, according to one illustrated embodiment.

FIG. 9 is a top plan view showing a portion of the optical system ofFIG. 8, illustrating a lateral shifting and change in focal distance inan image of a virtual object, according to one illustrated embodiment.

FIG. 10 is an elevational view showing a portion of an optical systemincluding a planar waveguide apparatus with a return planar waveguide,according to one illustrated embodiment.

FIG. 11 is an elevational view showing a portion of an optical systemincluding a planar waveguide apparatus with at least partiallyreflective mirrors or reflectors at opposed ends thereof to return lightthrough a planar waveguide, according to one illustrated embodiment.

FIG. 12 is a contour plot of a function for an exemplary diffractiveelement pattern, according to one illustrated embodiment.

FIGS. 13A-13E illustrate a relationship between a substrate index and afield of view, according to one illustrated embodiment.

FIG. 14 illustrates an internal circuitry of an exemplary AR system,according to one illustrated embodiment.

FIG. 15 illustrates hardware components of a head mounted AR system,according to one illustrated embodiment.

FIG. 16 illustrates an exemplary physical form of the head mounted ARsystem of FIG. 15.

FIG. 17 illustrates multiple user devices connected to each otherthrough a cloud server of the AR system.

FIG. 18 illustrates capturing 2D and 3D points in an environment of theuser, according to one illustrated embodiment.

FIG. 19 illustrates an overall system view depicting multiple AR systemsinteracting with a passable world model, according to one illustratedembodiment.

FIG. 20 is a schematic diagram showing multiple keyframes that captureand transmit data to the passable world model, according to oneillustrated embodiment.

FIG. 21 is a process flow diagram illustrating an interaction between auser device and the passable world model, according to one illustratedembodiment.

FIG. 22 is a process flow diagram illustrating recognition of objects byobject recognizers, according to one illustrated embodiment.

FIG. 23 is a schematic diagram illustrating a topological map, accordingto one illustrated embodiment.

FIG. 24 is a process flow diagram illustrating an identification of alocation of a user through the topological map of FIG. 23, according toone illustrated embodiment.

FIG. 25 is a schematic diagram illustrating a network of keyframes and apoint of stress on which to perform a bundle adjust, according to oneillustrated embodiment.

FIG. 26 is a schematic diagram that illustrates performing a bundleadjust on a set of keyframes, according to one illustrated embodiment.

FIG. 27 is a process flow diagram of an exemplary method of performing abundle adjust, according to one illustrated embodiment.

FIG. 28 is a schematic diagram illustrating determining new map pointsbased on a set of keyframes, according to one illustrated embodiment.

FIG. 29 is a process flow diagram of an exemplary method of determiningnew map points, according to one illustrated embodiment.

FIG. 30 is a system view diagram of an exemplary AR system, according toone illustrated embodiment.

FIG. 31 is a process flow diagram of an exemplary method of renderingvirtual content in relation to recognized objects, according to oneillustrated embodiment.

FIG. 32 is a plan view of another embodiment of the AR system, accordingto one illustrated embodiment.

FIG. 33 is a process flow diagram of an exemplary method of identifyingsparse and dense points, according to one illustrated embodiment.

FIG. 34 is a schematic diagram illustrating system components to projecttextured surfaces, according to one illustrated embodiment.

FIG. 35 is a plan view of an exemplary AR system illustrating aninteraction between cloud servers, error correction module and a machinelearning module, according to one illustrated embodiment.

FIGS. 36A-36I are schematic diagrams illustrating gesture recognition,according to one illustrated embodiment.

FIG. 37 is a process flow diagram of an exemplary method of performingan action based on a recognized gesture, according to one illustratedembodiment.

FIG. 38 is a plan view illustrating various finger gestures, accordingto one illustrated embodiment.

FIG. 39 is a process flow diagram of an exemplary method of determininguser input based on a totem, according to one illustrated embodiment.

FIG. 40 illustrates an exemplary totem in the form of a virtualkeyboard, according to one illustrated embodiment.

FIGS. 41A-41C illustrates another exemplary totem in the form of amouse, according to one illustrated embodiment.

FIGS. 42A-42C illustrates another exemplary totem in the form of a lotusstructure, according to one illustrated embodiment.

FIGS. 43A-43D illustrates other exemplary totems.

FIGS. 44A-44C illustrates exemplary totems in the form of rings,according to one illustrated embodiment.

FIGS. 45A-45C illustrates exemplary totems in the form of a hapticglove, a pen and a paintbrush, according to one illustrated embodiment.

FIGS. 46A-46B illustrated exemplary totems in the form of a keychain anda charm bracelet, according to one illustrated embodiment.

FIG. 47 is a process flow diagram of an exemplary method of generating avirtual user interface, according to one illustrated embodiment.

FIGS. 48A-48C illustrate various user interfaces through which tointeract with the AR system, according to the illustrated embodiments.

FIG. 49 is a process flow diagram of an exemplary method of constructinga customized user interface, according to one illustrated embodiment.

FIGS. 50A-50C illustrate users creating user interfaces, according toone illustrated embodiment.

FIGS. 51A-51C illustrate interacting with a user interface created inspace, according to one illustrated embodiment.

FIGS. 52A-52C are schematic diagrams illustrating creation of a userinterface on a palm of the user, according to one illustratedembodiment.

FIG. 53 is a process flow diagram of an exemplary method of retrievinginformation from the passable world model and interacting with otherusers of the AR system, according to one illustrated embodiment.

FIG. 54 is a process flow diagram of an exemplary method of retrievinginformation from a knowledge based in the cloud based on received input,according to one illustrated embodiment.

FIG. 55 is a process flow diagram of an exemplary method of recognizinga real-world activity, according to one illustrated embodiment.

FIGS. 56A-56B illustrate a user scenario of a user interacting with theAR system in an office environment, according to one illustratedembodiment.

FIG. 57 is another user scenario diagram illustrating creating an officeenvironment in the user's living room, according to one illustratedembodiment.

FIG. 58 is another user scenario diagram illustrating a user watchingvirtual television in the user's living room, according to oneillustrated embodiment.

FIG. 59 is another user scenario diagram illustrating the user of FIG.54 interacting with the virtual television through hand gestures,according to one illustrated embodiment.

FIGS. 60A-60B illustrates the user of FIGS. 58 and 59 interacting withthe AR system using other hand gestures, according to one illustratedembodiment.

FIGS. 61A-61E illustrate other applications opened by the user of FIGS.58-60 by interacting with various types of user interfaces, according toone illustrated embodiment.

FIGS. 62A-62D illustrate the user of FIGS. 58-61 changing a virtual skinof the user's living room, according to one illustrated embodiment.

FIG. 63 illustrates the user of FIGS. 58-61 using a totem to interactwith the AR system, according to one illustrated embodiment.

FIG. 64A-64B illustrates the user of FIGS. 58-63 using a physical objectas a user interface, according to one illustrated embodiment.

FIGS. 65A-65C illustrates the user of FIGS. 58-64 selecting a movie towatch on a virtual television screen, according to one illustratedembodiment.

FIGS. 66A-66J illustrate a user scenario of a mother and daughter on ashopping trip and interacting with the AR system, according to oneillustrated embodiment.

FIG. 67 illustrates another user scenario of a user browsing through avirtual bookstore, according to one illustrated embodiment.

FIGS. 68A-68F illustrates user scenario of using the AR system invarious healthcare and recreational settings, according to oneillustrated embodiment.

FIG. 69 illustrates yet another user scenario of a user interacting withthe AR system at a golf course, according to one illustrated embodiment.

DETAILED DESCRIPTION

Various embodiments will now be described in detail with reference tothe drawings, which are provided as illustrative examples of theinvention so as to enable those skilled in the art to practice theinvention. Notably, the figures and the examples below are not meant tolimit the scope of the present invention. Where certain elements of thepresent invention may be partially or fully implemented using knowncomponents (or methods or processes), only those portions of such knowncomponents (or methods or processes) that are necessary for anunderstanding of the present invention will be described, and thedetailed descriptions of other portions of such known components (ormethods or processes) will be omitted so as not to obscure theinvention. Further, various embodiments encompass present and futureknown equivalents to the components referred to herein by way ofillustration. Disclosed are methods and systems for generating virtualand/or augmented reality.

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various disclosedembodiments. However, one skilled in the relevant art will recognizethat embodiments may be practiced without one or more of these specificdetails, or with other methods, components, materials, etc. In otherinstances, well-known structures associated with computer systems,server computers, and/or communications networks have not been shown ordescribed in detail to avoid unnecessarily obscuring descriptions of theembodiments.

Unless the context requires otherwise, throughout the specification andclaims which follow, the word “comprise” and variations thereof, suchas, “comprises” and “comprising” are to be construed in an open,inclusive sense, that is as “including, but not limited to.”

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. Thus, the appearances of the phrases “in one embodiment” or“in an embodiment” in various places throughout this specification arenot necessarily all referring to the same embodiment. Furthermore, theparticular features, structures, or characteristics may be combined inany suitable manner in one or more embodiments.

As used in this specification and the appended claims, the singularforms “a,” “an,” and “the” include plural referents unless the contentclearly dictates otherwise. It should also be noted that the term “or”is generally employed in its sense including “and/or” unless the contentclearly dictates otherwise.

Numerous implementations are shown and described. To facilitateunderstanding, identical or similar structures are identified with thesame reference numbers between the various drawings, even though in someinstances these structures may not be identical.

The headings and Abstract of the Disclosure provided herein are forconvenience only and do not interpret the scope or meaning of theembodiments.

In contrast to the conventional approaches, at least some of the devicesand/or systems described herein enable: (1) a waveguide-based displaythat produces images at single optical viewing distance closer thaninfinity (e.g., arm's length); (2) a waveguide-based display thatproduces images at multiple, discrete optical viewing distances; and/or(3) a waveguide-based display that produces image layers stacked atmultiple viewing distances to represent volumetric 3D objects. Theselayers in the light field may be stacked closely enough together toappear continuous to the human visual system (i.e., one layer is withinthe cone of confusion of an adjacent layer). Additionally oralternatively, picture elements may be blended across two or more layersto increase perceived continuity of transition between layers in thelight field, even if those layers are more sparsely stacked (i.e., onelayer is outside the cone of confusion of an adjacent layer). Thedisplay system may be monocular or binocular.

Embodiments of the described volumetric 3D displays may advantageouslyallow digital content superimposed over the user's view of the realworld to be placed at appropriate viewing distances that do not requirethe user to draw his or her focus away from relevant real world objects.For example, a digital label or “call-out” for a real object can beplaced at the same viewing distance as that object, so both label andobject are in clear focus at the same time.

Embodiments of the described volumetric 3D displays may advantageouslyresult in stereoscopic volumetric 3D displays that mitigate or entirelyresolve the accommodation-vergence conflict produced in the human visualsystem by conventional stereoscopic displays. A binocular stereoscopicembodiment can produce 3D volumetric scenes in which the optical viewingdistance (i.e., the focal distance) matches the fixation distancecreated by the stereoscopic imagery—i.e., the stimulation to ocularvergence and ocular accommodation are matching, allowing users to pointtheir eyes and focus their eyes at the same distance.

FIG. 1 showing an optical system 100 including a primary waveguideapparatus 102, an optical coupler subsystem 104, and a control subsystem106, according to one illustrated embodiment.

The primary waveguide apparatus 102 includes one or more primary planarwaveguides 1 (only one show in FIG. 1), and one or more diffractiveoptical elements (DOEs) 2 associated with each of at least some of theprimary planar waveguides 1.

As best illustrated in FIG. 2, the primary planar waveguides 1 each haveat least a first end 108 a and a second end 108 b, the second end 108 bopposed to the first end 108 a along a length 110 of the primary planarwaveguide 1. The primary planar waveguides 1 each have a first face 112a and a second face 112 b, at least the first and the second faces 112a, 112 b (collectively 112) forming an at least partially internallyreflective optical path (illustrated by arrow 114 a and broken linearrow 114 b, collectively 114) along at least a portion of the length110 of the primary planar waveguide 1. The primary planar waveguide(s) 1may take a variety of forms which provides for substantially totalinternal reflection (TIR) for light striking the faces 112 at less thana defined critical angle. The planar waveguides 1 may, for example, takethe form of a pane or plane of glass, fused silica, acrylic, orpolycarbonate.

The DOEs 4 (illustrated in FIGS. 1 and 2 by dash-dot double line) maytake a large variety of forms which interrupt the TIR optical path 114,providing a plurality of optical paths (illustrated by arrows 116 a andbroken line arrows 116 b, collectively 116) between an interior 118 andan exterior 120 of the planar waveguide 1 extending along at least aportion of the length 110 of the planar waveguide 1. As explained belowin reference to FIGS. 3A-3C, the DOEs 4 may advantageously combine thephase functions of a linear diffraction grating with that of a circularor radial symmetric lens, allowing positioning of apparent objects andfocus plane for apparent objects. Such may be achieved on aframe-by-frame, subframe-by-subframe, or even pixel-by-pixel basis.

With reference to FIG. 1, the optical coupler subsystem 104 opticallycouples light to, or from, the waveguide apparatus 102. As illustratedin FIG. 1, the optical coupler subsystem may include an optical element5, for instance a reflective surface, mirror, dichroic mirror or prismto optically couple light to, or from, an edge 122 of the primary planarwaveguide 1. The optical coupler subsystem 104 may additionally oralternatively include a collimation element 6 that collimates light.

The control subsystem 106 includes one or more light sources 11 anddrive electronics 12 that generate image data that is encoded in theform of light that is spatially and/or temporally varying. As notedabove, a collimation element 6 may collimate the light, and thecollimated light optically s coupled into one or more primary planarwaveguides 1 (only one illustrated in FIGS. 1 and 2).

As illustrated in FIG. 2, the light propagates along the primary planarwaveguide with at least some reflections or “bounces” resulting from theTIR propagation. It is noted that some implementations may employ one ormore reflectors in the internal optical path, for instance thin-films,dielectric coatings, metalized coatings, etc., which may facilitatereflection. Light propagates along the length 110 of the waveguide 1intersects with one or more DOEs 4 at various positions along the length110.

As explained below in reference to FIGS. 4A-4C, the DOE(s) 4 may beincorporated within the primary planar waveguide 1 or abutting oradjacent one or more of the faces 112 of the primary planar waveguide 1.The DOE(s) 4 accomplishes at least two functions. The DOE(s) 4 shift anangle of the light, causing a portion of the light to escape TIR, andemerge from the interior 118 to the exterior 120 via one or more faces112 of the primary planar waveguide 1. The DOE(s) 4 focus theout-coupled light at one or more viewing distances. Thus, someonelooking through a face 112 a of the primary planar waveguide 1 can seedigital imagery at one or more viewing distances.

FIG. 3A shows a linear diffraction or diffractive phase function 300,according to one illustrated embodiment. The linear diffraction ordiffractive function 300 may be that of a linear diffractive grating,for example a Bragg grating.

FIG. 3B showings a radially circular or radially symmetric lens phasefunction 310, according to one illustrated embodiment.

FIG. 3B shows a phase pattern 320 for at least one diffractive opticalelement that combines the linear diffraction and the radially circularlens functions 300, 310, according to one illustrated embodiment, atleast one diffractive optical element associated with at least oneplanar waveguide. Notably, each band has a curved wavefront.

While FIGS. 1 and 2 show the DOE 2 positioned in the interior 118 of theprimary planar waveguide 1, spaced from the faces 112, the DOE 2 may bepositioned at other locations in other implementations, for example asillustrated in FIGS. 4A-4C.

FIG. 4A shows a waveguide apparatus 102 a including a primary planarwaveguide 1 and at least one DOE 2 carried on an outer surface or face112 of the primary planar waveguide 1, according to one illustratedembodiment. For example, the DOE 2 may be deposited on the outer surfaceor face 112 of the primary planar waveguide 1, for instance as apatterned metal layer.

FIG. 4B shows a waveguide apparatus 102 b including a primary planarwaveguide 1 and at least one DOE 2 positioned internally immediatelyadjacent an outer surface or face 112 of the primary planar waveguide 1,according to one illustrated embodiment. For example, the DOE 2 may beformed in the interior 118 via selective or masked curing of material ofthe primary planar waveguide 1. Alternatively, the DOE 2 may be adistinct physical structure incorporated into the primary planarwaveguide 1.

FIG. 4C shows a waveguide apparatus 102 c including a primary planarwaveguide 1 and at least one DOE 2 formed in an outer surface of theprimary planar waveguide 1, according to one illustrated embodiment. TheDOE 2 may, for example be etched, patterned, or otherwise formed in theouter surface or face 112 of the primary planar waveguide 1, forinstances as grooves. For example, the DOE 2 may take the form of linearor saw tooth ridges and valleys which may be spaced at one or moredefined pitches (i.e., space between individual elements or featuresextending along the length 110). The pitch may be a linear function ormay be a non-linear function.

The primary planar waveguide 1 is preferably at least partiallytransparent. Such allows one or more viewers to view the physicalobjects (i.e., the real world) on a far side of the primary planarwaveguide 1 relative to a vantage of the viewer. This may advantageouslyallow viewers to view the real world through the waveguide andsimultaneously view digital imagery that is relayed to the eye(s) by thewaveguide.

In some implementations a plurality of waveguides systems may beincorporated into a near-to-eye display. For example, a plurality ofwaveguides systems may be incorporated into a head-worn, head-mounted,or helmet-mounted display—or other wearable display.

In some implementations, a plurality of waveguides systems may beincorporated into a head-up display (HUD), that is not worn (e.g., anautomotive HUD, avionics HUD). In such implementations, multiple viewersmay look at a shared waveguide system or resulting image field. Multipleviewers may, for example see or optically perceive a digital or virtualobject from different viewing perspectives that match each viewer'srespective locations relative to the waveguide system.

The optical system 100 is not limited to use of visible light, but mayalso employ light in other portions of the electromagnetic spectrum(e.g., infrared, ultraviolet) and/or may employ electromagneticradiation that is outside the band of “light” (i.e., visible, UV, orIR), for example employing electromagnetic radiation or energy in themicrowave or X-ray portions of the electromagnetic spectrum.

In some implementations, a scanning light display is used to couplelight into a plurality of primary planar waveguides. The scanning lightdisplay can comprise a single light source that forms a single beam thatis scanned over time to form an image. This scanned beam of light may beintensity-modulated to form pixels of different brightness levels.Alternatively, multiple light sources may be used to generate multiplebeams of light, which are scanned either with a shared scanning elementor with separate scanning elements to form imagery.

These light sources may comprise different wavelengths, visible and/ornon-visible, they may comprise different geometric points of origin (X,Y, or Z), they may enter the scanner(s) at different angles ofincidence, and may create light that corresponds to different portionsof one or more images (flat or volumetric, moving or static).

The light may, for example, be scanned to form an image with a vibratingoptical fiber, for example as discussed in U.S. patent application Ser.No. 13/915,530, International Patent Application Serial No.PCT/US2013/045267, and U.S. provisional patent application Ser. No.61/658,355. The optical fiber may be scanned biaxially by apiezoelectric actuator. Alternatively, the optical fiber may be scanneduniaxially or triaxially. As a further alternative, one or moreoptically components (e.g., rotating polygonal reflector or mirror,oscillating reflector or mirror) may be employed to scan an output ofthe optical fiber.

The optical system 100 is not limited to use in producing images or asan image projector or light field generation. For example, the opticalsystem 100 or variations thereof may optical, be employed as an imagecapture device, such as a digital still or digital moving image captureor camera system.

FIG. 5A shows an optical system 500 including a waveguide apparatus, anoptical coupler subsystem to optically couple light to or from thewaveguide apparatus, and a control subsystem, according to oneillustrated embodiment.

Many of the structures of the optical system 500 of FIG. 5A are similaror even identical to those of the optical system 100 of FIG. 1. In theinterest of conciseness, in many instances only significant differencesare discussed below.

The optical system 500 may employ a distribution waveguide apparatus, torelay light along a first axis (vertical or Y-axis in view of FIG. 5A),and expand the light's effective exit pupil along the first axis (e.g.,Y-axis). The distribution waveguide apparatus, may, for example includea distribution planar waveguide 3 and at least one DOE 4 (illustrated bydouble dash-dot line) associated with the distribution planar waveguide3. The distribution planar waveguide 3 may be similar or identical in atleast some respects to the primary planar waveguide 1, having adifferent orientation therefrom. Likewise, the at least one DOE 4 may besimilar or identical in at least some respects to the DOE 2. Forexample, the distribution planar waveguide 3 and/or DOE 4 may becomprised of the same materials as the primary planar waveguide 1 and/orDOE 2, respectively

The relayed and exit-pupil expanded light is optically coupled from thedistribution waveguide apparatus into one or more primary planarwaveguide 1. The primary planar waveguide 1 relays light along a secondaxis, preferably orthogonal to first axis, (e.g., horizontal or X-axisin view of FIG. 5A). Notably, the second axis can be a non-orthogonalaxis to the first axis. The primary planar waveguide 1 expands thelight's effective exit pupil along that second axis (e.g. X-axis). Forexample, a distribution planar waveguide 3 can relay and expand lightalong the vertical or Y-axis, and pass that light to the primary planarwaveguide 1 which relays and expands light along the horizontal orX-axis.

FIG. 5B shows the optical system 500, illustrating generation thereby ofa single focus plane that is capable of being positioned closer thanoptical infinity.

The optical system 500 may include one or more sources of red, green,and blue laser light 11, which may be optically coupled into a proximalend of a single mode optical fiber 9. A distal end of the optical fiber9 may be threaded or received through a hollow tube 8 of piezoelectricmaterial. The distal end protrudes from the tube 8 as fixed-freeflexible cantilever 7. The piezoelectric tube 8 is associated with 4quadrant electrodes (not illustrated). The electrodes may, for example,be plated on the outside, outer surface or outer periphery or diameterof the tube 8. A core electrode (not illustrated) is also located in acore, center, inner periphery or inner diameter of the tube 8.

Drive electronics 12, for example electrically coupled via wires 11,drive opposing pairs of electrodes to bend the piezoelectric tube 8 intwo axes independently. The protruding distal tip of the optical fiber 7has mechanical modes of resonance. The frequencies of resonance whichdepend upon a diameter, length, and material properties of the opticalfiber 7. By vibrating the piezoelectric tube 8 near a first mode ofmechanical resonance of the fiber cantilever 7, the fiber cantilever 7is caused to vibrate, and can sweep through large deflections.

By stimulating resonant vibration in two axes, the tip of the fibercantilever 7 is scanned biaxially in an area filling 2D scan. Bymodulating an intensity of light source(s) 11 in synchrony with the scanof the fiber cantilever 7, light emerging from the fiber cantilever 7forms an image. Descriptions of such a set up are provide in U.S. patentapplication Ser. No. 13/915,530, International Patent Application SerialNo. PCT/US2013/045267, and U.S. provisional patent application Ser. No.61/658,355, all of which are incorporated by reference herein in theirentireties.

A component of an optical coupler subsystem 104 collimates the lightemerging from the scanning fiber cantilever 7. The collimated light isreflected by mirrored surface 5 into a narrow distribution planarwaveguide 3 which contains at least one diffractive optical element(DOE) 4. The collimated light propagates vertically (i.e., relative toview of FIG. 5B) along the distribution planar waveguide 3 by totalinternal reflection, and in doing so repeatedly intersects with the DOE4. The DOE 4 preferably has a low diffraction efficiency. This causes afraction (e.g., 10%) of the light to be diffracted toward an edge of thelarger primary planar waveguide 1 at each point of intersection with theDOE 4, and a fraction of the light to continue on its originaltrajectory down the length of the distribution planar waveguide 3 viaTIR.

At each point of intersection with the DOE 4, additional light isdiffracted toward the entrance of the primary waveguide 1. By dividingthe incoming light into multiple outcoupled sets, the exit pupil of thelight is expanded vertically by the DOE 4 in the distribution planarwaveguide 3. This vertically expanded light coupled out of distributionplanar waveguide 3 enters the edge of the primary planar waveguide 1.

Light entering primary waveguide 1 propagates horizontally (i.e.,relative to view of FIG. 5B) along the primary waveguide 1 via TIR. Asthe light intersects with DOE 2 at multiple points as it propagateshorizontally along at least a portion of the length of the primarywaveguide 1 via TIR. The DOE 2 may advantageously be designed orconfigured to have a phase profile that is a summation of a lineardiffraction grating and a radially symmetric diffractive lens. The DOE 2may advantageously have a low diffraction efficiency.

At each point of intersection between the propagating light and the DOE2, a fraction of the light is diffracted toward the adjacent face of theprimary waveguide 1 allowing the light to escape the TIR, and emergefrom the face of the primary waveguide 1. The radially symmetric lensaspect of the DOE 2 additionally imparts a focus level to the diffractedlight, both shaping the light wavefront (e.g., imparting a curvature) ofthe individual beam as well as steering the beam at an angle thatmatches the designed focus level. FIG. 5B illustrates four beams 18, 19,20, 21 extending geometrically to a focus point 13, and each beam isadvantageously imparted with a convex wavefront profile with a center ofradius at focus point 13 to produce an image or virtual object 22 at agiven focal plane.

FIG. 5C shows the optical system 500 illustrating generation thereby ofa multi-focal volumetric display, image or light field. The opticalsystem 500 may include one or more sources of red, green, and blue laserlight 11, optically coupled into a proximal end of a single mode opticalfiber 9. A distal end of the optical fiber 9 may be threaded or receivedthrough a hollow tube 8 of piezoelectric material. The distal endprotrudes from the tube 8 as fixed-free flexible cantilever 7. Thepiezoelectric tube 8 is associated with 4 quadrant electrodes (notillustrated). The electrodes may, for example, be plated on the outsideor outer surface or periphery of the tube 8. A core electrode (notillustrated) is positioned in a core, center, inner surface, innerperiphery or inner diameter of the tube 8.

Drive electronics 12, for example coupled via wires 11, drive opposingpairs of electrodes to bend the piezoelectric tube 8 in two axesindependently. The protruding distal tip of the optical fiber 7 hasmechanical modes of resonance. The frequencies of resonance of whichdepend upon the a diameter, length, and material properties of the fibercantilever 7. By vibrating the piezoelectric tube 8 near a first mode ofmechanical resonance of the fiber cantilever 7, the fiber cantilever 7is caused to vibrate, and can sweep through large deflections.

By stimulating resonant vibration in two axes, the tip of the fibercantilever 7 is scanned biaxially in an area filling 2D scan. Bymodulating the intensity of light source(s) 11 in synchrony with thescan of the fiber cantilever 7, the light emerging from the fibercantilever 7 forms an image. Descriptions of such a set up are providein U.S. patent application Ser. No. 13/915,530, International PatentApplication Serial No. PCT/US2013/045267, and U.S. provisional patentapplication Ser. No. 61/658,355, all of which are incorporated byreference herein in their entireties.

A component of an optical coupler subsystem 104 collimates the lightemerging from the scanning fiber cantilever 7. The collimated light isreflected by mirrored surface 5 into a narrow distribution planarwaveguide 3, which contains diffractive optical element (DOE) 4. Thecollimated light propagates along the distribution planar waveguide bytotal internal reflection (TIR), and in doing so repeatedly intersectswith the DOE 4. The DOE has a low diffraction efficiency.

This causes a fraction (e.g., 10%) of the light to be diffracted towardan edge of a larger primary planar waveguide 1 at each point ofintersection with the DOE 4, and a fraction of the light to continue onits original trajectory down the distribution planar waveguide 3 viaTIR. At each point of intersection with the DOE 4, additional light isdiffracted toward the entrance of the primary planar waveguide 1. Bydividing the incoming light into multiple out-coupled sets, the exitpupil of the light is expanded vertically by DOE 4 in distributionplanar waveguide 3. This vertically expanded light coupled out of thedistribution planar waveguide 3 enters the edge of the primary planarwaveguide 1.

Light entering primary waveguide 1 propagates horizontally (i.e.,relative to view of FIG. 5C) along the primary waveguide 1 via TIR. Asthe light intersects with DOE 2 at multiple points as it propagateshorizontally along at least a portion of the length of the primarywaveguide 1 via TIR. The DOE 2 may advantageously be designed orconfigured to have a phase profile that is a summation of a lineardiffraction grating and a radially symmetric diffractive lens. The DOE 2may advantageously have a low diffraction efficiency. At each point ofintersection between the propagating light and the DOE 2, a fraction ofthe light is diffracted toward the adjacent face of the primarywaveguide 1 allowing the light to escape the TIR, and emerge from theface of the primary waveguide 1.

The radially symmetric lens aspect of the DOE 2 additionally imparts afocus level to the diffracted light, both shaping the light wavefront(e.g., imparting a curvature) of the individual beam as well as steeringthe beam at an angle that matches the designed focus level. FIG. 5Cillustrates a first set of four beams 18, 19, 20, 21 extendinggeometrically to a focus point 13, and each beam 18, 19, 20, 21 isadvantageously imparted with a convex wavefront profile with a center ofradius at focus point 13 to produce another portion of the image orvirtual object 22 at a respective focal plane. FIG. 5C illustrates asecond set of four beams 24, 25, 26, 27 extending geometrically to afocus point 23, and each beam 24, 25, 26, 27 is advantageously impartedwith a convex wavefront profile with a center of radius at focus point23 to produce another portion of the image or virtual object 22 at arespective focal plane.

FIG. 6 shows an optical system 600, according to one illustratedembodiment. The optical system 600 is similar in some respects to theoptical systems 100, 500. In the interest of conciseness, only some ofthe difference are discussed.

The optical system 600 includes a waveguide apparatus 102, which asdescribed above may comprise one or more primary planar waveguides 1 andassociated DOE(s) 2 (not illustrated in FIG. 6). In contrast to theoptical system 500 of FIGS. 5A-5C, the optical system 600 employs aplurality of microdisplays or projectors 602 a-602 e (only five shown,collectively 602) to provide respective image data to the primary planarwaveguide(s) 1. The microdisplays or projectors 602 are generallyarrayed or arranged along are disposed along an edge 122 of the primaryplanar waveguide 1.

There may, for example, be a one to one (1:1) ratio or correlationbetween the number of planar waveguides 1 and the number ofmicrodisplays or projectors 602. The microdisplays or projectors 602 maytake any of a variety of forms capable of providing images to theprimary planar waveguide 1. For example, the microdisplays or projectors602 may take the form of light scanners or other display elements, forinstance the cantilevered fiber 7 previously described. The opticalsystem 600 may additionally or alternatively include a collimationelement 6 that collimates light provided from microdisplay or projectors602 prior to entering the primary planar waveguide(s) 1.

The optical system 600 can enable the use of a single primary planarwaveguide 1, rather using two or more primary planar waveguides 1 (e.g.,arranged in a stacked configuration along the Z-axis of FIG. 6). Themultiple microdisplays or projectors 602 can be disposed, for example,in a linear array along the edge 122 of a primary planar waveguide thatis closest to a temple of a viewer's head. Each microdisplay orprojector 602 injects modulated light encoding sub-image data into theprimary planar waveguide 1 from a different respective position, thusgenerating different pathways of light.

These different pathways can cause the light to be coupled out of theprimary planar waveguide 1 by a multiplicity of DOEs 2 at differentangles, focus levels, and/or yielding different fill patterns at theexit pupil. Different fill patterns at the exit pupil can bebeneficially used to create a light field display. Each layer in thestack or in a set of layers (e.g., 3 layers) in the stack may beemployed to generate a respective color (e.g., red, blue, green). Thus,for example, a first set of three adjacent layers may be employed torespectively produce red, blue and green light at a first focal depth. Asecond set of three adjacent layers may be employed to respectivelyproduce red, blue and green light at a second focal depth. Multiple setsmay be employed to generate a full 3D or 4D color image field withvarious focal depths.

FIG. 7 shows a planar waveguide apparatus 700 including a planarwaveguide 1 with a plurality of DOEs 2 a-2 d (four illustrated, each asa double dash-dot line, collectively 2), according to one illustratedembodiment.

The DOEs 2 are stacked along an axis 702 that is generally parallel tothe field-of-view of the planar waveguide 700. While illustrated as allbeing in the interior 118, in some implementations one, more or even allof the DOEs may be on an exterior of the planar waveguide 1.

In some implementations, each DOE 2 may be capable of beingindependently switched ON and OFF. That is each DOE 2 can be made activesuch that the respective DOE 2 diffracts a significant fraction of lightthat intersects with the respective DOE 2, or it can be renderedinactive such that the respective DOE 2 either does not diffract lightintersecting with the respective DOE 2 at all, or only diffracts aninsignificant fraction of light. “Significant” in this context meansenough light to be perceived by the human visual system when coupled outof the planar waveguide 1, and “insignificant” means not enough light tobe perceived by the human visual system, or a low enough level to beignored by a viewer.

The switchable DOEs 2 may be switched on one at a time, such that onlyone DOE 2 in the primary planar waveguide 1 is actively diffracting thelight in the primary planar waveguide 1, to emerge from one or morefaces 112 of the primary planar waveguide 1 in a perceptible amount.Alternatively, two or more DOEs 2 may be switched ON simultaneously,such that their diffractive effects are combined.

The phase profile of each DOE 2 is advantageously a summation of alinear diffraction grating and a radially symmetric diffractive lens.Each DOE 2 preferably has a low (e.g., less than 50%) diffractionefficiency.

The light intersects with the DOEs at multiple points along the lengthof the planar waveguide 1 as the light propagates horizontally in theplanar waveguide 1 via TIR. At each point of intersection between thepropagating light and a respective one of the DOEs 2, a fraction of thelight is diffracted toward the adjacent face 112 of the planar waveguide1, allowing the light to escape TIR and emerge from the face 112 of theplanar waveguide 1.

The radially symmetric lens aspect of the DOE 2 additionally imparts afocus level to the diffracted light, both shaping the light wavefront(e.g., imparting a curvature) of the individual beam, as well assteering the beam at an angle that matches the designed focus level.Such is best illustrated in FIG. 5B where the four beams 18, 19, 20, 21,if geometrically extended from the far face 112 b of the planarwaveguide 1, intersect at a focus point 13, and are imparted with aconvex wavefront profile with a center of radius at focus point 13.

Each DOE 2 in the set of DOEs can have a different phase map. Forexample, each DOE 2 can have a respective phase map such that each DOE2, when switched ON, directs light to a different position in X, Y, orZ. The DOEs 2 may, for example, vary from one another in their lineargrating aspect and/or their radially symmetric diffractive lens aspect.If the DOEs 2 vary from one another in their diffractive lens aspect,different DOEs 2 (or combinations of DOEs 2) will produce sub-images atdifferent optical viewing distances—i.e., different focus distances.

If the DOEs 2 vary from one another in their linear grating aspect,different DOEs 2 will produce sub-images that are shifted laterallyrelative to one another. Such lateral shifts can be beneficially used tocreate a foveated display, to steer a display image with non-homogenousresolution or other non-homogenous display parameters (e.g., luminance,peak wavelength, polarization, etc.) to different lateral positions, toincrease the size of the scanned image, to produce a variation in thecharacteristics of the exit pupil, and/or to generate a light fielddisplay. Lateral shifts may be advantageously employed to preform tilingor realize a tiling effect in generated images.

For example, a first DOE 2 in the set, when switched ON, may produce animage at an optical viewing distance of 1 meter (e.g., focal point 23 inFIG. 5C) for a viewer looking into the primary or emission face 112 a ofthe planar waveguide 1. A second DOE 2 in the set, when switched ON, mayproduce an image at an optical viewing distance of 1.25 meters (e.g.,focal point 13 in FIG. 5C) for a viewer looking into the primary oremission face 112 a of the planar waveguide 1.

By switching exemplary DOEs 2 ON and OFF in rapid temporal sequence(e.g., on a frame-by-frame basis, a sub-frame basis, a line-by-linebasis, a sub-line basis, pixel-by-pixel basis, or sub-pixel-by-sub-pixelbasis) and synchronously modulating the image data being injected intothe planar waveguide 1, for instance by a scanning fiber displaysub-system, a composite multi-focal volumetric image is formed that isperceived to a be a single scene to the viewer. By rendering differentobjects or portions of objects to sub-images relayed to the eye of theviewer (at location 22 in FIG. 5C) by the different DOEs 2, virtualobjects or images are placed at different optical viewing distances, ora virtual object or image can be represented as a 3D volume that extendsthrough multiple planes of focus.

FIG. 8 shows a portion of an optical system 800 including a plurality ofplanar waveguide apparati 802 a-802 d (four shown, collectively 802),according to one illustrated embodiment.

The planar waveguide apparati 802 are stacked, arrayed, or arrangedalong an axis 804 that is generally parallel to the field-of-view of theportion of the optical system 800. Each of the planar waveguide apparati802 includes at least one planar waveguide 1 (only one called out inFIG. 8) and at least one associated DOE 2 (illustrated by dash-dotdouble line, only one called out in FIG. 8). While illustrated as allbeing in the interior 118, in some implementations one, more or even allof the DOEs 2 may be on an exterior of the planar waveguide 1.Additionally or alternatively, while illustrated with a single lineararray of DOEs 2 per planar waveguide 1, one or more of the planarwaveguides 1 may include two or more stacked, arrayed or arranged DOEs2, similar to the implementation described with respect to FIG. 7.

Each of the planar waveguide apparati 802 a-802 d may functionanalogously to the operation of the DOEs 2 of the optical system 7 (FIG.7), That is the DOEs 2 of the respective planar waveguide apparati 802may each have a respective phase map, the phase maps of the various DOEs2 being different from one another. While dynamic switching (e.g.,ON/OFF) of the DOEs 2 was employed in the optical system 700 (FIG. 7),such can be avoided in the optical system 800. Instead of, or inadditional to dynamic switching, the optical system 800 may selectivelyroute light to the planar waveguide apparati 802 a-802 d based on therespective phase maps. Thus, rather than turning ON a specific DOE 2having a desired phase map, the optical system 800 may route light to aspecific planar waveguide 802 that has or is associated with a DOE 2with the desired phase mapping. Again, the may be in lieu of, or inaddition to, dynamic switching of the DOEs 2.

In one example, the microdisplays or projectors may be selectivelyoperated to selectively route light to the planar waveguide apparati 802a-802 d based on the respective phase maps. In another example, each DOE4 may be capable of being independently switched ON and OFF, similar toas explained with reference to switching DOEs 2 ON and OFF. The DOEs 4may be switched ON and OFF to selectively route light to the planarwaveguide apparati 802 a-802 d based on the respective phase maps.

FIG. 8 also illustrated outward emanating rays from two of the planarwaveguide apparati 802 a, 802 d. For sake of illustration, a first oneof the planar waveguide apparatus 802 a produces a plane or flatwavefront (illustrated by flat lines 804 about rays 806, only oneinstance of each called out for sake of drawing clarity) at an infinitefocal distance. In contrast, another one of the planar waveguideapparatus 802 d produces a convex wavefront (illustrated by arc 808about rays 810, only one instance of each called out for sake of drawingclarity) at a defined focal distance less than infinite (e.g., 1 meter).

As illustrated in FIG. 9, the planar waveguide apparati 802 a-802 d maylaterally shift the appearance and/or optical viewing distances—i.e.,different focus distances of a virtual object 900 a-900 c with respectto an exit pupil 902.

FIG. 10 shows a portion of an optical system 1000 including a planarwaveguide apparatus 102 with a return planar waveguide 1002, accordingto one illustrated embodiment.

The planar waveguide apparatus 102 may be similar to those describedherein, for example including one or more planar waveguides 1 and one ormore associated DOEs 2.

In contrast to previously described implementations, the optical system1000 includes the return planar waveguide 1002, which provides a TIRoptical path for light to return from one end 108 b of the planarwaveguide 1 to the other end 108 a of the planar waveguide 1 forrecirculation. The optical system 1000 also include is a first mirror orreflector 1004, located at a distal end 108 a (i.e., end opposed to endat which light first enters). The mirror or reflector 1004 at the distalend 108 a may be completely reflecting. The optical system 1000optionally includes is a second mirror or reflector 1006, located at aproximate end 108 b (i.e., end at which light first enters as indicatedby arrow 1010). The second mirror or reflector 1006 may be a dichroicmirror or prism, allowing light to initially enter the optical system,and then reflecting light returned from the distal end 108 a.

Thus, light may enter at the proximate end 108 b as indicated by arrow1010. The light may traverse or propagate along the planar waveguide 1in a first pass, as illustrated by arrow 1012, exiting at the distal end112 b. The first mirror or reflector 1004 may reflect the light topropagate via the return planar waveguide 1002, as illustrated by arrow1014. The second mirror or reflector 1006 may reflect the remaininglight back to the planar waveguide 1 for a second pass, as illustratedby arrow 1016. This may repeat until there is no appreciable light leftto recirculate. This recirculation of light may advantageously increaseluminosity or reduce system luminosity requirements.

FIG. 11 shows a portion of an optical system 1100 including a planarwaveguide apparatus 102 with at least partially reflective mirrors orreflectors 1102 a, 1102 b at opposed ends 112 a, 112 b thereof to returnlight through a planar waveguide 1, according to one illustratedembodiment.

Light may enter at the proximate end 108 b as indicated by arrow 1110.The light may traverse or propagate along the planar waveguide 1 in afirst pass, as illustrated by arrow 1112, exiting at the distal end 112b. The first mirror or reflector 1102 a may reflect the light topropagate the planar waveguide 1, as illustrated by arrow 1114. Thesecond mirror or reflector 1006 may optionally reflect the remaininglight back to the planar waveguide 1 for a second pass (notillustrated). This may repeat until there is no appreciable light leftto recirculate. This recirculation of light may advantageously increaseluminosity or reduce system luminosity requirements.

In some implementations, an optical coupling system collimates the lightemerging from a multiplicity of displays or projectors, prior tooptically coupling the light to a planar waveguide. This opticalcoupling system may include, but is not limited to, a multiplicity ofDOEs, refractive lenses, curved mirrors, and/or freeform opticalelements. The optical coupling subsystem may serve multiple purposes,such as collimating the light from the multiplicity of displays andcoupling the light into a waveguide. The optical coupling subsystem mayinclude a mirrored surface or prism to reflect or deflect the collimatedlight into a planar waveguide.

In some implementations the collimated light propagates along a narrowplanar waveguide via TIR, and in doing so repeatedly intersects with amultiplicity of DOEs 2. As described above, the DOEs 2 may comprise orimplement respective different phase maps, such that the DOEs 2 steerthe light in the waveguide along respective different paths. Forexample, if the multiple DOEs 2 contain linear grating elements withdifferent pitches, the light is steered at different angles, which maybeneficially be used to create a foveated display, steer anon-homogenous display laterally, increase the lateral dimensions of theout-coupled image, increase effective display resolution by interlacing,generate different fill patterns at the exit pupil, and/or generate alight field display.

As previously described, a multiplicity of DOEs 2 may be arrayed orarranged or configured in a stack within or on a respective planarwaveguide 1, 3.

The DOEs 2 in the distribution planar waveguide 3 may have a lowdiffraction efficiency, causing a fraction of the light to be diffractedtoward the edge of the larger primary planar waveguide 1, at each pointof intersection, and a fraction of the light to continue on its originaltrajectory down the distribution planar waveguide 3 via TIR. At eachpoint of intersection, additional light is diffracted toward an edge orentrance of the primary planar waveguide 1. By dividing the incominglight into multiple out-coupled sets, the exit pupil of the light isexpanded vertically by multiplicity of DOEs 4 in distribution planarwaveguide 3.

As described above, vertically expanded light coupled out of thedistribution planar waveguide 3 enters an edge of larger primary planarwaveguide 1, and propagates horizontally along the length of the primaryplanar waveguide 1 via TIR.

The multiplicity of DOEs 4 in the narrow distribution planar waveguide 3can have a low diffraction efficiency, causing a fraction of the lightto be diffracted toward the edge of the larger primary planar waveguide1 at each point of intersection, and a fraction of the light to continueon its original trajectory down the distribution planar waveguide 3 byTIR. At each point of intersection, additional light is diffractedtoward the entrance of larger primary planar waveguide 1. By dividingthe incoming light into multiple out-coupled sets, the exit pupil of thelight is expanded vertically by the multiplicity of DOEs 4 indistribution planar waveguide 3. A low diffraction efficiency in themultiplicity of DOEs in the primary planar waveguide 1 enables viewersto see through the primary planar waveguide 1 to view real objects, witha minimum of attenuation or distortion.

In at least one implementation, the diffraction efficiency of themultiplicity of DOEs 2 is low enough to ensure that any distortion ofreal world is not perceptible to a human looking through the waveguideat the real world.

Since a portion or percentage of light is diverted from the internaloptical path as the light transits the length of the planar waveguide(s)1, 3, less light may be diverted from one end to the other end of theplanar waveguide 1, 3 if the diffraction efficiency is constant alongthe length of the planar waveguide 1,3. This change or variation inluminosity or output across the planar waveguide 1, 3 is typicallyundesirable. The diffraction efficiency may be varied along the lengthto accommodate for this undesired optical effect. The diffractionefficiency may be varied in a fixed fashion, for example by fixedlyvarying a pitch of the DOEs 2, 4 along the length when the DOEs 2, 4and/or planar waveguide 1, 3 is manufactured or formed. Intensity oflight output may be advantageously be increased or varied as a functionof lateral offset of pixels in the display or image.

Alternatively, the diffraction efficiency may be varied dynamically, forexample by fixedly varying a pitch of the DOEs 2, 4 along the lengthwhen the DOEs 2, 4 and/or planar waveguide 1,3 is in use. Such mayemploy a variety of techniques, for instance varying an electricalpotential or voltage applied to a material (e.g., liquid crystal). Forexample, voltage changes could be applied, for instance via electrodes,to liquid crystals dispersed in a polymer host or carrier medium.

The voltage may be used to change the molecular orientation of theliquid crystals to either match or not match a refractive index of thehost or carrier medium. As explained herein, a structure which employs astack or layered array of switchable layers (e.g., DOEs 2, planerwaveguides 1), each independently controllable may be employed toadvantageous affect.

In at least one implementation, the summed diffraction efficiency of asubset of simultaneously switched on DOEs 2 of the multiplicity of DOEs2 is low enough to enable viewers to see through the waveguide to viewreal objects, with a minimum of attenuation or distortion.

It may be preferred if the summed diffraction efficiency of a subset ofsimultaneously switched on DOEs 2 of the multiplicity of DOEs 2 is lowenough to ensure that any distortion of real world is not perceptible toa human looking through the waveguide at the real world.

As described above, each DOE 2 in the multiplicity or set of DOEs 2 maybe capable of being switched ON and OFF—i.e., it can be made active suchthat the respective DOE 2 diffracts a significant fraction of light thatintersects with the respective DOE 2, or can be rendered inactive suchthat the respective DOE 2 either does not diffract light intersectingwith it at all, or only diffracts an insignificant fraction of light.“Significant” in this context means enough light to be perceived by thehuman visual system when coupled out of the waveguide, and“insignificant” means not enough light to be perceived by the humanvisual system, or a low enough level to be ignored by a viewer.

The switchable multiplicity of DOEs 2 may be switched ON one at a time,such that only one DOE 2 associated with the large primary planarwaveguide 1 is actively diffracting the light in the primary planarwaveguide 1 to emerge from one or more faces 112 of the primary planarwaveguide 1 in a perceptible amount. Alternatively, two or more DOEs 2in the multiplicity of DOEs 2 may be switched ON simultaneously, suchthat their diffractive effects are advantageously combined. It may thusbe possible to realize 2N combinations, where N is the number of DOEs 2in associated with a respective planar waveguide 1, 3.

In at least some implementations, the phase profile or map of each DOE 2in at least the large or primary planar waveguide 1 is or reflects asummation of a linear diffraction grating and a radially symmetricdiffractive lens, and has a low (less than 50%) diffraction efficiency.Such is illustrated in FIGS. 3A-3C. In particular, the hologram phasefunction comprises a linear function substantially responsible forcoupling the light out of the waveguide, and a lens functionsubstantially responsible for creating a virtual image

  p(x, y) = p 1(x, y) + p 2(x, y),  where$\mspace{20mu}{{{p\; 1( {x,y} )} = \frac{x\; 0\; y\; 1\; y}{nr}},\mspace{20mu}{and}}$${p\; 2( {x,y} )} = {{x\; 2\; y\; 0( \frac{x}{nr} )^{2}} + {x\; 2\; y\; 2( \frac{x}{nr} )^{2}( \frac{y}{nr} )^{2}} + {x\; 2\; y\; 4( \frac{x}{nr} )^{2}( \frac{y}{nr} )^{4}} + {x\; 4\; y\; 0( \frac{x}{nr} )^{4}} + {x\; 4\; y\; 2( \frac{x}{nr} )^{4}( \frac{y}{nr} )^{2}} + {x\; 6\; y\; 0( \frac{x}{nr} )^{6}} + {x\; 0\; y\; 2( \frac{y}{nr} )^{2}} + {x\; 0\; y\; 4( \frac{y}{nr} )^{4}} + {x\; 0\; y\; 6( \frac{y}{nr} )^{6}}}$

In this example, the coefficients of p2 are constrained to produce aradially symmetric phase function.

An example EDGE element was designed for a 40 degree diagonal field ofview having a 16×9 aspect ratio. The virtual object distance is 500 mm(2 diopters). The design wavelength is 532 nanometers. The substratematerial is fused silica, and the y angles of incidence in the substratelie between 45 and 72 degrees. The y angle of incidence required togenerate an on axis object at is 56 degrees. The phase function definingthe example element is:

${\Phi\; g} = {\frac{12.4113\; x^{2}}{{mm}^{2}} - \frac{0.00419117x^{4}}{{mm}^{4}} - \frac{14315.y}{mm} - \frac{12.4113\; y^{2}}{{mm}^{2}} - \frac{0.00838233\; x^{2}y^{2}}{{mm}^{4}} - \frac{0.00419117\; y^{4}}{{mm}^{4}}}$

The diffractive element pattern is generated by evaluating the 2 piphase contours. FIG. 12 shows a contour plot 4000 illustrating thefunction evaluated over a 20×14 mm element area (required to provide a 4mm eye box at a 25 mm eye relief. The contour interval was chosen tomake the groove pattern visible. The actual groove spacing in thisdesign is approximately 0.5 microns.

The relationship between substrate index and field of view is describedin FIGS. 13A-13E. The relationship is non-trivial, but a highersubstrate index always allows for a large field of view. One shouldalways prefer higher index of refraction materials if all otherconsiderations are equal.

Referring to FIG. 13A, plot 4002 describes a relationship between thesubstrate index and field of view according to one embodiment. Referringto the following equation,

$k_{j} = \frac{2\;\pi}{\lambda_{j}}$where j is the region index. The index 0 is used to indicate free space(air).

k₂d sin (θ₂) − k₁d sin (θ₁) = m 2 π${{\frac{2\;\pi}{\lambda_{1}}{\sin( \theta_{2} )}} - {\frac{2\;\pi}{\lambda_{1}}{\sin( \theta_{1} )}}} = {m\frac{2\;\pi}{d}}$${\frac{2\;\pi}{\lambda_{2}}{\sin( \theta_{2} )}} = {{m\frac{2\;\pi}{d}} + {\frac{2\;\pi}{\lambda_{1}}{\sin( \theta_{1} )}}}$${k_{2}{\sin( \theta_{2} )}} = {{m\frac{2\;\pi}{d}} + {k_{1}{\sin( \theta_{1} )}}}$$k_{2\; y} = {{m\frac{2\;\pi}{d}} + k_{1\; y}}$k_(2 y) = mk_(g) + k_(1 y)

Alternative formulation normalized using the free space wavelength maybe the following:

${\overset{harpoonup}{h}}_{j} = \frac{{\overset{harpoonup}{k}}_{j}}{k_{0}}$$h_{j} = {\frac{k_{j}}{k_{0}} = n_{j}}$$h_{g} = {\frac{k_{g}}{k_{0}} = \frac{\lambda_{0}}{d}}$h_(2 y) = mh_(g) + h_(1 y), where  h_(jy) = h_(j)sin (θ_(j))

If ′|h_(2y)|≤h₂, then the wave associated with {right arrow over (h₂)}(vector h2) is not evanescent.

For the substrate guided wave, the rectangle in the following diagramindicates the region of allowed projections of h (vector h) into the X Yplane. The outer circle has radius n, and indicates a wave vectorparallel to the X Y plane. The inner circle has radius 1 and indicatesthe TIR (total internal reflection) boundary.

Referring now to FIG. 13 B (plot 4004) in the normalized representation,(vector h) is a vector of magnitude n independent of free spacewavelength. When the index is 1, the components are the direction ofcosines of {right arrow over (k)} (vector k).k _(x) ² +k _(y) ² +k _(z) ² =k ₀ ²h _(x) ² +h _(y) ² +h _(z) ² =n ²

The wavelengths used to design an earlier fiber scanner lens (ref.sfe-06aa.zmx) were 443, 532, and 635 nm. The red and blue wavelengthsare used in the following calculation.

Referring now to FIG. 13C-13E, FIGS. 13C-13E show plots (4006-4010) ofnormalized wave vector regions projected into the x y plane (i.e.parallel to the substrate). The rectangle in the middle represents theeye field of view. The top two rectangles represent the waveguide vectorprojections required to produce the eye field of view. The arrowsindicate the deflection provided by the grating.

The unit radius circle represents the TIR (total internal reflection)constraint for a guided wave in the substrate, and the 1.5 radius circlerepresents a wave propagating parallel to the substrate when the indexn=1.5. Wave vectors propagating between the two circles are allowed.This plot is for the substrate oriented vertically, a 50° diagonal (16×9format) eye field of view, and a 0.36 micron grating line spacing. Notethat the rectangle in the concentric circle lies inside the region ofallowed region, whereas the topmost rectangle lies in the evanescentregion.

By increasing the groove spacing to 5.2 microns, the vector from theouter circle (red) can be brought inside the allowed region, but then amajority of the vectors in the concentric circle (blue) do not totallyinternally reflect (FIG. 13 D)

Tilting the substrate with respect to the eye is equivalent to biasingthe eye field of view with respect to the substrate. This plot shows theeffect of tilting the waveguide 45° and increasing the groove width to0.85 mm. Note that the difference between the grating arrows is less,and that both the vectors fall substantially within the allowed region(FIG. 13E).

First order diffraction efficiencies should be in the neighborhood of0.01 to 0.20. Lower values require higher input energy to createspecified image brightness, while larger values lead to increased pupilnon-uniformity. The particular value chosen depends on the particularapplication requirements.

It may be advantageous to vary one or more characteristics of the DOEs2, for example along a longitudinal or axial dimension thereof. Forinstance, a pitch may be varied, or a height of a groove or angle (e.g.,90 degree, 60 degree) of a structure forming the DOE 2 or portionthereof. Such may advantageously address higher order aberrations.

Two beams of mutually coherent light may be employed to dynamically varythe properties of the DOEs 2. The beams of mutually coherent light may,for example, be generated via a single laser and a beam splitter. Thebeams may interact with a liquid crystal film to create a highinterference pattern on or in the liquid crystal film to dynamicallygenerate at least one diffraction element, e.g., a grating such as aBragg grating. The DOEs 2 may be addressable on a pixel-by-pixel basis.Thus, for example, a pitch of the elements of the DOEs 2 may be varieddynamically. The interference patterns are typically temporary, but maybe held sufficiently long to affect the diffraction of light.

Further, diffraction gratings may be employed to split lateral chromaticaberrations. For example, a relative difference in angle can be expectedfor light of different colors when passed through a DOE 2. Where a pixelis being generated via three different colors, the colors may not beperceived as being in the same positions due to the difference inbending of the respective colors of light. This may be addressed byintroducing a very slight delay between the signals used to generateeach color for any given pixel. One way of addressing this is viasoftware, where image data is “pre-misaligned” or pre-wrapped, toaccommodate the differences in location of the various colors making upeach respective pixel. Thus, the image data for generating a bluecomponent of a pixel in the image may be offset spatially and/ortemporally with respect to a red component of the pixel to accommodate aknown or expected shift due to diffraction. Likewise, a green componentmay be offset spatially and/or temporally with respect to a red and bluecomponents of the pixel.

The image field may be generated to have a higher concentration of lightor image information proximal to the viewer in contrast to portions thatare relatively distal to the viewer. Such may advantageously take intoaccount the typically higher sensitivity of the vision system forrelative close objects or images as compared to more distal objects ofimages. Thus, virtual objects in the foreground of an image field may berendered at a higher resolution (e.g., higher density of focal planes)than objects in the background of the image field. The variousstructures and approaches described herein advantageously allow suchnon-uniform operation and generation of the image field.

In at least some implementations, the light intersects with themultiplicity of DOEs 2 at multiple points as it propagates horizontallyvia TIR. At each point of intersection between the propagating light andthe multiplicity of DOEs 2, a fraction of the light is diffracted towardthe adjacent face of the planar waveguide 1, 3 allowing the light toescape TIR and emerge from the face 112 of the planar waveguide 1, 3.

In at least some implementations, the radially symmetric lens aspect ofthe DOE 2 additionally imparts a focus level to the diffracted light,both shaping the light wavefront (e.g., imparting a curvature) of theindividual beam as well as steering the beam at an angle that matchesthe designed focus level. In FIG. 5B, the four beams 18, 19, 20, 21, ifgeometrically extended from the far face of the primary planar waveguide1, intersect at a focus point 13, and are imparted with a convexwavefront profile with a center of radius at focus point 13.

In at least some implementations, each DOE 2 in the multiplicity or setof DOEs 2 can have a different phase map, such that each DOE 2, whenswitched ON or when fed light, directs light to a different position inX, Y, or Z. The DOEs 2 may vary from one another in their linear gratingaspect and/or their radially symmetric diffractive lens aspect. If theDOEs 2 vary in their diffractive lens aspect, different DOEs 2 (orcombinations of DOEs) will produce sub-images at different opticalviewing distances—i.e., different focus distances. If the DOEs 2 vary intheir linear grating aspect, different DOEs 2 will produce sub-imagesthat are shifted laterally relative to one another.

In at least some implementations, lateral shifts generated by themultiplicity of DOEs can be beneficially used to create a foveateddisplay. In at least some implementations, lateral shifts generated bythe multiplicity of DOEs 2 can be beneficially used to steer a displayimage with non-homogenous resolution or other non-homogenous displayparameters (e.g., luminance, peak wavelength, polarization, etc.) todifferent lateral positions. In at least some implementations, lateralshifts generated by the multiplicity of DOEs can be beneficially used toincrease the size of the scanned image.

In at least some implementations, lateral shifts generated by themultiplicity of DOEs can be beneficially used to produce a variation inthe characteristics of the exit pupil. In at least some implementations,lateral shifts generated by the multiplicity of DOEs can be beneficiallyused, to produce a variation in the characteristics of the exit pupiland generate a light field display.

In at least some implementations, a first DOE 2, when switched ON, mayproduce an image at a first optical viewing distance 23 (FIG. 5C) for aviewer looking into the face of the primary planar waveguide 1. A secondDOE 2 in the multiplicity, when switched ON, may produce an image at asecond optical viewing distance 13 (FIG. 5C) for a viewer looking intothe face of the waveguide.

In at least some implementations, DOEs 2 are switched ON and OFF inrapid temporal sequence. In at least some implementations, DOEs 2 areswitched ON and OFF in rapid temporal sequence on a frame-by-framebasis. In at least some implementations, DOEs 2 are switched ON and OFFin rapid temporal sequence on a sub-frame basis. In at least someimplementations, DOEs 2 are switched ON and OFF in rapid temporalsequence on a line-by-line basis.

In at least some implementations, DOEs 2 are switched ON and OFF inrapid temporal sequence on a sub-line basis. In at least someimplementations, DOEs 2 are switched ON and OFF in rapid temporalsequence on a pixel-by-pixel basis. In at least some implementations,DOEs 2 are switched ON and OFF in rapid temporal sequence on asub-pixel-by-sub-pixel basis. In at least some implementations, DOEs 2are switched ON and OFF in rapid temporal sequence on some combinationof a frame-by-frame basis, a sub-frame basis, a line-by-line basis, asub-line basis, pixel-by-pixel basis, and/or sub-pixel-by-sub-pixelbasis.

In at least some implementations, while DOEs 2 are switched ON and OFFthe image data being injected into the waveguide by the multiplicity ofmicrodisplays is simultaneously modulated. In at least someimplementations, while DOEs 2 are switched ON and OFF the image databeing injected into the waveguide by the multiplicity of microdisplaysis simultaneously modulated to form a composite multi-focal volumetricimage that is perceived to a be a single scene to the viewer.

In at least some implementations, by rendering different objects orportions of objects to sub-images relayed to the eye (position 22 inFIG. 5C) by the different DOEs 2, objects are placed at differentoptical viewing distances, or an object can be represented as a 3Dvolume that extends through multiple planes of focus.

In at least some implementations, the multiplicity of switchable DOEs 2is switched at a fast enough rate to generate a multi-focal display thatis perceived as a single scene.

In at least some implementations, the multiplicity of switchable DOEs 2is switched at a slow rate to position a single image plane at a focaldistance. The accommodation state of the eye is measured and/orestimated either directly or indirectly. The focal distance of thesingle image plane is modulated by the multiplicity of switchable DOEsin accordance with the accommodative state of the eye. For example, ifthe estimated accommodative state of the eye suggests that the viewer isfocused at a 1 meter viewing distance, the multiplicity of DOEs isswitched to shift the displayed image to approximate at 1 meter focusdistance. If the eye's accommodative state is estimated to have shiftedto focus at, e.g., a 2 meter viewing distance, the multiplicity of DOEs2 is switched to shift the displayed image to approximate at 2 meterfocus distance.

In at least some implementations, the multiplicity of switchable DOEs 2is switched at a slow rate to position a single image plane at a focaldistance. The accommodation state of the eye is measured and/orestimated either directly or indirectly. The focal distance of thesingle image plane is modulated by the multiplicity of switchable DOEsin accordance with the accommodative state of the eye, and the imagedata presented by the multiplicity of display elements is switchedsynchronously.

For example, if the estimated accommodative state of the eye suggeststhat the viewer is focused at a 1 meter viewing distance, themultiplicity of DOEs 2 is switched to shift the displayed image toapproximate at 1 meter focus distance, and the image data is updated torender the virtual objects at a virtual distance of 1 meter in sharpfocus and to render virtual objects at a virtual distance other than 1meter with some degree of blur, with greater blur for objects fartherfrom the 1 meter plane.

If the eye's accommodative state is estimated to have shifted to focusat, e.g., a 2 meter viewing distance, the multiplicity of DOEs isswitched to shift the displayed image to approximate at 2 meter focusdistance and the image data is updated to render the virtual objects ata virtual distance of 2 meters in sharp focus and to render virtualobjects at a virtual distance other than 2 meters with some degree ofblur, with greater blur for objects farther from the 2 meter plane.

In at least some implementations, the DOEs 2 may be used to bias raysoutwardly to create a large field of view, at least up to a limit atwhich light leaks from the planar waveguide(s) 1. For example, varying apitch of a grating may achieve a desired change in angle sufficient tomodify the angles associated with or indicative of a field of view. Insome implements, pitch may be tuned to achieve a lateral or side-to-sidemovement or scanning motion along at least one lateral (e.g., Y-axis).Such may be done in two dimensions to achieve a lateral or side-to-sidemovement or scanning motion along both the Y-axis and X-axis. One ormore acousto-optic modulators may be employed, changing frequency,period, or angle of deflection.

Various standing surface wave techniques (e.g., standing plane wavefield) may be employed, for example to dynamically adjust thecharacteristics of the DOEs 2. For instance standing waves may begenerated in a liquid crystal medium trapped between two layers,creating an interference pattern with desired frequency, wavelengthand/or amplitude characteristics.

The DOEs 2 may be arranged to create a toe in effect, creating an eyebox that tapers from larger to smaller as the light approaches theviewer from the planar waveguide 1. The light box may taper in one ortwo dimensions (e.g., Y-axis, X-axis, as function of position along theZ-axis). Concentrating light may advantageously reduce luminosityrequires or increase brightness. The light box should still be maintainsufficiently large to accommodate expected eye movement.

While various embodiments have located the DOEs 2 in or on the primaryplanar waveguide 1, other implementations may located one or more DOEs 2spaced from the primary planar waveguide 1. For example, a first set ofDOEs 2 may be positioned between the primary planar waveguide 1 and theviewer, spaced from the primary planar waveguide 1. Additionally, asecond set of DOEs 2 may be positioned between the primary planarwaveguide 1 and background or real world, spaced from the primary planarwaveguide 1. Such may be used to cancel light from the planar waveguideswith respect to light from the background or real world, in somerespects similar to noise canceling headphones.

The various embodiments described above can be combined to providefurther embodiments. To the extent that they are not inconsistent withthe specific teachings and definitions herein, all of the U.S. patents,U.S. patent application publications, U.S. patent applications, foreignpatents, foreign patent applications and non-patent publicationsreferred to in this specification and/or listed in the Application DataSheet, including but not limited to U.S. patent application Ser. No.13/915,530, International Patent Application Serial No.PCT/US2013/045267, and U.S. provisional patent application Ser. No.61/658,355, are incorporated herein by reference, in their entirety.Aspects of the embodiments can be modified, if necessary, to employsystems, circuits and concepts of the various patents, applications andpublications to provide yet further embodiments.

System Components

The DOEs described above may be incorporated into an augmented reality(AR) system. The DOE elements or volumetric 3D displays allow for thecreation of multiple focal planes based on which numerous virtualreality or augmented virtual reality applications may be realized.Methods and systems of the overall AR system will be described. Variousapplications of the AR system will also be described further below. Itshould be appreciated that the systems below may use the volumetric 3Ddisplays in their optical components, or any other suitable opticalcomponents (e.g., birdbath optics, free form optics, etc.) may besimilarly used. The AR system may be a stationary system or a portablesystem that may have a body or head worn component. For illustrativepurposes, the following discussion will focus on portable AR systems,but it should be appreciated that stationary systems may also be used.

FIG. 14 shows an architecture 1000 for the electronics for a body orhead worn component, according to one illustrated embodiment. It shouldbe appreciated that the following system architecture may be used foroptical elements apart from volumetric 3D displays.

The body or head worn component may include one or more printed circuitboard components, for instance left and right printed circuit boardassemblies (PCBA). As illustrated, the left PCBA includes most of theactive electronics, while the right PCBA supports principally supportsthe display or projector elements.

The right PCBAs may include a number of projector driver structureswhich provide image information and control signals to image generationcomponents. For example, the right PCBA may carry a first or leftprojector driver structure and a second or right projector driverstructure. The first or left projector driver structure join a first orleft projector fiber and a set of signal lines (e.g., piezo driverwires).

The second or right projector driver structure join a second or rightprojector fiber and a set of signal lines (e.g., piezo driver wires).The first or left projector driver structure is communicatively coupledto a first or left image projector, while the second or right projectordrive structure is communicatively coupled to the second or right imageprojector.

In operation, the image projectors render virtual content to the leftand right eyes (e.g., retina) of the user via respective opticalcomponents (e.g., the volumetric 3D display described above, forexample), for instance waveguides and/or compensation lenses. The imageprojectors may, for example, include left and right projectorassemblies. The projector assemblies may use a variety of differentimage forming or production technologies, for example, fiber scanprojectors, liquid crystal displays (LCD), digital light processing(DLP) displays.

Where a fiber scan projector is employed, images may be delivered alongan optical fiber, to be projected therefrom via a tip of the opticalfiber (e.g., as shown in FIG. 1). The tip may be oriented to feed intothe waveguide. An end of the optical fiber with the tip from whichimages project may be supported to flex or oscillate. A number ofpiezoelectric actuators may control an oscillation (e.g, frequency,amplitude) of the tip. The projector driver structures provide images torespective optical fiber and control signals to control thepiezoelectric actuators, to project images to the user's eyes.

Continuing with the right PCBA, a button board connector may providecommunicative and physical coupling a button board which carries varioususer accessible buttons, keys, switches or other input devices. Theright PCBA may include a right earphone or speaker connector, tocommunicatively couple audio signals to a right earphone or speaker ofthe head worn component. The right PCBA may also include a rightmicrophone connector to communicatively couple audio signals from amicrophone of the head worn component. The right PCBA may furtherinclude a right occlusion driver connector to communicatively coupleocclusion information to a right occlusion display of the head worncomponent. The right PCBA may also include a board-to-board connector toprovide communications with the left PCBA via a board-to-board connectorthereof.

The right PCBA may be communicatively coupled to one or more rightoutward facing or world view cameras which are body or head worn, andoptionally a right cameras visual indicator (e.g., LED) whichilluminates to indicate to others when images are being captured. Theright PCBA may be communicatively coupled to one or more right eyecameras, carried by the head worn component, positioned and orientatedto capture images of the right eye to allow tracking, detection, ormonitoring of orientation and/or movement of the right eye. The rightPCBA may optionally be communicatively coupled to one or more right eyeilluminating sources (e.g., LEDs), which as explained herein,illuminates the right eye with a pattern (e.g., temporal, spatial) ofillumination to facilitate tracking, detection or monitoring oforientation and/or movement of the right eye.

The left PCBA may include a control subsystem, which may include one ormore controllers (e.g., microcontroller, microprocessor, digital signalprocessor, graphical processing unit, central processing unit,application specific integrated circuit (ASIC), field programmable gatearray (FPGA), and/or programmable logic unit (PLU)). The control systemmay include one or more non-transitory computer- or processor readablemedium that stores executable logic or instructions and/or data orinformation. The non-transitory computer- or processor readable mediummay take a variety of forms, for example volatile and nonvolatile forms,for instance read only memory (ROM), random access memory (RAM, DRAM,SD-RAM), flash memory, etc. The non-transitory computer- or processorreadable medium may be formed as one or more registers, for example of amicroprocessor, FPGA or ASIC.

The left PCBA may include a left earphone or speaker connector, tocommunicatively couple audio signals to a left earphone or speaker ofthe head worn component. The left PCBA may include an audio signalamplifier (e.g., stereo amplifier), which is communicative coupled tothe drive earphones or speakers The left PCBA may also include a leftmicrophone connector to communicatively couple audio signals from amicrophone of the head worn component. The left PCBA may further includea left occlusion driver connector to communicatively couple occlusioninformation to a left occlusion display of the head worn component.

The left PCBA may also include one or more sensors or transducers whichdetect, measure, capture or otherwise sense information about an ambientenvironment and/or about the user. For example, an accelerationtransducer (e.g., three axis accelerometer) may detect acceleration inthree axis, thereby detecting movement. A gyroscopic sensor may detectorientation and/or magnetic or compass heading or orientation. Othersensors or transducers may be employed,

The left PCBA may be communicatively coupled to one or more left outwardfacing or world view cameras which are body or head worn, and optionallya left cameras visual indicator (e.g., LED) which illuminates toindicate to others when images are being captured. The left PCBA may becommunicatively coupled to one or more left eye cameras, carried by thehead worn component, positioned and orientated to capture images of theleft eye to allow tracking, detection, or monitoring of orientationand/or movement of the left eye. The left PCBA may optionally becommunicatively coupled to one or more left eye illuminating sources(e.g., LEDs), which as explained herein, illuminates the left eye with apattern (e.g., temporal, spatial) of illumination to facilitatetracking, detection or monitoring of orientation and/or movement of theleft eye.

The PCBAs are communicatively coupled with the distinct computationcomponent (e.g., belt pack) via one or more ports, connectors and/orpaths. For example, the left PCBA may include one or more communicationsports or connectors to provide communications (e.g., bi-directionalcommunications) with the belt pack. The one or more communications portsor connectors may also provide power from the belt pack to the left PCBAThe left PCBA may include power conditioning circuitry (e.g., DC/DCpower converter, input filter), electrically coupled to thecommunications port or connector and operable to condition (e.g., stepup voltage, step down voltage, smooth current, reduce transients).

The communications port or connector may, for example, take the form ofa data and power connector or transceiver (e.g., Thunderbolt® port, USB®port). The right PCBA may include a port or connector to receive powerfrom the belt pack. The image generation elements may receive power froma portable power source (e.g., chemical battery cells, primary orsecondary battery cells, ultra-capacitor cells, fuel cells), which may,for example be located in the belt pack.

As illustrated, the left PCBA includes most of the active electronics,while the right PCBA supports principally supports the display orprojectors, and the associated piezo drive signals. Electrical and/orfiber optic connections are employed across a front, rear or top of thebody or head worn component.

Both PCBAs may be communicatively (e.g., electrically, optically)coupled to a belt pack. It should be appreciated that other embodimentsof the AR system may not include a belt back, and the associatedcircuitry of the belt pack may simply be incorporated in a compact forminto the electronics of the head worn component of the AR system.

The left PCBA includes the power subsystem and a high speedcommunications subsystem. The right PCBA handles the fiber display piezodrive signals. In the illustrated embodiment, only the right PCBA needsto be optically connected to the belt pack.

While illustrated as employing two PCBAs, the electronics of the body orhead worn component may employ other architectures. For example, someimplementations may use a fewer or greater number of PCBAs. Also forexample, various components or subsystems may be arranged differentlythan illustrated in FIG. 14. For example, in some alternativeembodiments some of the components illustrated in FIG. 14 as residing onone PCBA, may be located on the other PCBA, without loss of generality.

As illustrated, each individual may use their own respective AR system.In some implementations, the respective AR systems may communicatebetween one another. For example, two or more proximately located ARsystems may communicate between one another. As described furtherherein, communications may occur after performance of a handshakingprotocol. The AR systems may communicate wirelessly via one or moreradios. As discussed above, such radios may be capable of short rangedirect communications, or may be capable of longer range directcommunications (i.e., without a repeater, extender, etc.). Additionallyor alternatively, indirect longer range communications may be achievedvia one or more intermediary devices (e.g., wireless access points,repeaters, extenders).

The head-worn component, some of whose components, including circuitry,have been described above, has many components, including opticalcomponents, camera systems etc. that enable a user of the system toenjoy 3D vision.

Referring to FIG. 15, one embodiment of the head-worn AR system has asuitable user display device (14) as shown in FIG. 15. The user displaydevice may comprise a display lens (82) which may be mounted to a user'shead or eyes by a housing or frame (84). The display lens (82) maycomprise one or more transparent mirrors positioned by the housing (84)in front of the user's eyes (20) and configured to bounce projectedlight (38) into the eyes (20) and facilitate beam shaping, while alsoallowing for transmission of at least some light from the localenvironment in an augmented reality configuration (in a virtual realityconfiguration, it may be desirable for the display system to be capableof blocking substantially all light from the local environment, such asby a darkened visor, blocking curtain, all black LCD panel mode, or thelike).

It should be appreciated that various optical systems may be used as asuitable display lens. In one embodiment, the volumetric 3D display,discussed above, may be used as the display lens in this exemplarysystem.

In the depicted embodiment, two wide-field-of-view machine visioncameras (16) are coupled to the housing (84) to image the environmentaround the user; in one embodiment these cameras (16) are dual capturevisible light/infrared light cameras. The depicted embodiment alsocomprises a pair of scanned-laser shaped-wavefront (i.e., for depth)light projector modules with display mirrors and optics configured toproject light (38) into the eyes (20) as shown. The depicted embodimentalso comprises two miniature infrared cameras (24) paired with infraredlight sources (26, such as light emitting diodes “LED” s), which areconfigured to be able to track the eyes (20) of the user to supportrendering and user input.

The system (14) further features a sensor assembly (39), which maycomprise X, Y, and Z axis accelerometer capability as well as a magneticcompass and X, Y, and Z axis gyro capability, preferably providing dataat a relatively high frequency, such as 200 Hz. The depicted system (14)also comprises a head pose processor (36), such as an ASIC (applicationspecific integrated circuit), FPGA (field programmable gate array),and/or ARM processor (advanced reduced-instruction-set machine), whichmay be configured to calculate real or near-real time user head posefrom wide field of view image information output from the capturedevices (16). Also shown is another processor (32) configured to executedigital and/or analog processing to derive pose from the gyro, compass,and/or accelerometer data from the sensor assembly (39).

The depicted embodiment also features a GPS (37, global positioningsatellite) subsystem to assist with pose and positioning. Finally, thedepicted embodiment comprises a rendering engine (34) which may featurehardware running a software program configured to provide renderinginformation local to the user to facilitate operation of the scannersand imaging into the eyes of the user, for the user's view of the world.The rendering engine (34) is operatively coupled (81, 70, 76/78, 80;i.e., via wired or wireless connectivity) to the sensor pose processor(32), the image pose processor (36), the eye tracking cameras (24), andthe projecting subsystem (18) such that light of rendered augmentedand/or virtual reality objects is projected using a scanned laserarrangement (18) in a manner similar to a retinal scanning display. Thewavefront of the projected light beam (38) may be bent or focused tocoincide with a desired focal distance of the augmented and/or virtualreality object.

The mini infrared cameras (24) may be utilized to track the eyes tosupport rendering and user input (i.e., where the user is looking, whatdepth he is focusing; as discussed below, eye verge may be utilized toestimate depth of focus). The GPS (37), gyros, compass, andaccelerometers (39) may be utilized to provide course and/or fast poseestimates. The camera (16) images and pose, in conjunction with datafrom an associated cloud computing resource, may be utilized to map thelocal world and share user views with a virtual or augmented realitycommunity. While much of the hardware in the display system (14)featured in FIG. 14 is depicted directly coupled to the housing (84)which is adjacent the display (82) and eyes (20) of the user, thehardware components depicted may be mounted to or housed within othercomponents, such as a belt-mounted component, as discussed above.

In one embodiment, all of the components of the system (14) featured inFIG. 15 are directly coupled to the display housing (84) except for theimage pose processor (36), sensor pose processor (32), and renderingengine (34), and communication between the latter three and theremaining components of the system (14) may be by wirelesscommunication, such as ultra wideband, or wired communication. Thedepicted housing (84) preferably is head-mounted and wearable by theuser. It may also feature speakers, such as those which may be insertedinto the ears of a user and utilized to provide sound to the user whichmay be pertinent to an augmented or virtual reality experience, andmicrophones, which may be utilized to capture sounds local to the user.

Regarding the projection of light (38) into the eyes (20) of the user,in one optional embodiment the mini cameras (24) may be utilized tomeasure where the centers of a user's eyes (20) are geometrically vergedto, which, in general, coincides with a position of focus, or “depth offocus”, of the eyes (20). A 3-dimensional surface of all points the eyesverge to is called the “horopter”. The focal distance may take on afinite number of depths, or may be infinitely varying. Light projectedfrom the vergence distance appears to be focused to the subject eye(20), while light in front of or behind the vergence distance isblurred.

Further, it has been discovered that spatially coherent light with abeam diameter of less than about 0.7 millimeters is correctly resolvedby the human eye regardless of where the eye focuses; given thisunderstanding, to create an illusion of proper focal depth, the eyevergence may be tracked with the mini cameras (24), and the renderingengine (34) and projection subsystem (18) may be utilized to render allobjects on or close to the horopter in focus, and all other objects atvarying degrees of defocus (i.e., using intentionally-created blurring).A see-through light guide optical element configured to project coherentlight into the eye may be provided by suppliers such as Lumus, Inc.

Preferably the system renders to the user at a frame rate of about 60frames per second or greater. As described above, preferably the minicameras (24) may be utilized for eye tracking, and software may beconfigured to pick up not only vergence geometry but also focus locationcues to serve as user inputs. Preferably such system is configured withbrightness and contrast suitable for day or night use. In one embodimentsuch system preferably has latency of less than about 20 millisecondsfor visual object alignment, less than about 0.1 degree of angularalignment, and about 1 arc minute of resolution, which is approximatelythe limit of the human eye. The display system (14) may be integratedwith a localization system, which may involve the GPS element, opticaltracking, compass, accelerometer, and/or other data sources, to assistwith position and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (i.e., such information would facilitate the glasses toknow where they are with respect to the real world).

Other suitable display device include but are not limited to desktop andmobile computers, smartphones, smartphones which may be enhancedadditional with software and hardware features to facilitate or simulate3-D perspective viewing (for example, in one embodiment a frame may beremovably coupled to a smartphone, the frame featuring a 200 Hz gyro andaccelerometer sensor subset, two small machine vision cameras with widefield of view lenses, and an ARM processor—to simulate some of thefunctionality of the configuration featured in FIG. 15), tabletcomputers, tablet computers which may be enhanced as described above forsmartphones, tablet computers enhanced with additional processing andsensing hardware, head-mounted systems that use smartphones and/ortablets to display augmented and virtual viewpoints (visualaccommodation via magnifying optics, mirrors, contact lenses, or lightstructuring elements), non-see-through displays of light emittingelements (LCDs, OLEDs, vertical-cavity-surface-emitting lasers, steeredlaser beams, etc.), see-through displays that simultaneously allowhumans to see the natural world and artificially generated images (forexample, light-guide optical elements, transparent and polarized OLEDsshining into close-focus contact lenses, steered laser beams, etc.),contact lenses with light-emitting elements (such as those availablefrom Innovega, Inc., of Bellevue, Wash., under the tradename Loptik®;they may be combined with specialized complimentary eyeglassescomponents), implantable devices with light-emitting elements, andimplantable devices that stimulate the optical receptors of the humanbrain.

Now that the circuitry and the basic components of the AR system, andspecifically the user display portion of the system has been described,various physical forms of the head worn component of the AR system willbe described briefly.

Referring now to FIG. 16, an exemplary embodiment of a physical form ofthe head worn component of the AR system will be briefly described inrelation to the overall AR system. As shown in FIG. 16, the head worncomponent comprises optics coupled with a user display system thatallows the user to view virtual or augmented reality content. The lightassociated with the virtual content, when projected to the user displaysystem of the head worn component, may appear to be coming from variousfocal depths, giving the user a sense of 3D perception.

It should be appreciated, as will be described in further detail below,that the head worn component of the AR system or the belt pack of the ARsystem, also shown in FIG. 16, are connectively coupled to one or morenetworks such that the AR system is constantly retrieving and uploadinginformation to the cloud. For example, the virtual content beingprojected to the user through the display system may be associated withvirtual content downloaded from the cloud. Or, in other embodiment,images captured through the user's FOV cameras may be processed anduploaded to the cloud, such that another user may be able to experiencethe physical surroundings of the first user, as if the other user werephysically present along with the first user. More user scenarios suchas the above will be described further below.

As shown in FIG. 16, the head worn component 1002 may simply resemble apair of reading glasses or goggles, or in other embodiments, may takethe form of a helmet display, or any other form factor. The belt pack istypically communicatively coupled to one or both sides of the head worncomponent, as explained above.

Cloud Servers

FIG. 17 illustrates a communications architecture which employs one ormore hub, central, or distributed, server computer systems 280 and oneor more individual AR systems 208 communicatively coupled by one or morewired or wireless networks 204, according to one illustrated embodiment.

The server computer systems 280 may, for example, be clustered. Forinstance, clusters of server computer systems may be located at variousgeographically dispersed locations. Such may facilitate communications,shortening transit paths and/or provide for redundancy.

Specific instances of personal AR systems 208 may be communicativelycoupled to the server computer system(s). The server computer system(s)may maintain information about a specific user's own physical and/orvirtual worlds. The server computer system(s) 280 may allow a given userto share information about the specific user's own physical and/orvirtual worlds with other users. Additionally or alternatively, theserver computer system(s) 280 may allow other users to share informationabout their own physical and/or virtual worlds with the given orspecific user. As described herein, server computer system(s) 280 mayallow mapping and/or characterizations of large portions of the physicalworlds.

Information may be collected via the personal AR system of one or moreusers. The models of the physical world may be developed over time, andby collection via a large number of users. This may allow a given userto enter a new portion or location of the physical world, yet benefit byinformation collected by others who either previously or are currentlyin the particular location. Models of virtual worlds may be created overtime via user by a respective user.

The personal AR system(s) 208 may be communicatively coupled to theserver computer system(s). For example, the personal AR system(s) may bewirelessly communicatively coupled to the server computer system(s) viaone or more radios. The radios may take the form of short range radios,as discussed above, or relatively long range radios, for examplecellular chip sets and antennas. The personal AR system(s) willtypically be communicatively coupled to the server computer system(s)indirectly, via some intermediary communications network or component.For instance, the personal AR system(s) will typically becommunicatively coupled to the server computer system(s) 280 via one ormore telecommunications provider systems, for example one or morecellular communications provider networks.

Other Components

In many implementations, the AR system may include other components. TheAR system or Sensorywear™ augmented reality devices may, for example,include one or more haptic devices or components. The haptic device(s)or component(s) may be operable to provide a tactile sensation to auser. For example, the haptic device(s) or component(s) may provide atactile sensation of pressure and/or texture when touching virtualcontent (e.g., virtual objects, virtual tools, other virtualconstructs). The tactile sensation may replicate a feel of a physicalobject which a virtual object represents, or may replicate a feel of animagined object or character (e.g., a dragon) which the virtual contentrepresents.

In some implementations, haptic devices or components may be worn by theuser. An example of a haptic device in the form of a user wearable gloveis described herein. In some implementations, haptic devices orcomponents may be held the user. An example of a haptic device in theform of a user wearable glove and as is described herein. Other examplesof haptic devices in the form of various haptic totems are describedherein. The AR system may additionally or alternatively employ othertypes of haptic devices or components.

The AR system may, for example, include one or more physical objectswhich are manipulable by the user to allow input or interaction with theAR system. These physical objects are referred to herein as totems. Sometotems may take the form of inanimate objects, for example a piece ofmetal or plastic, a wall, a surface of table. Alternatively, some totemsmay take the form of animate objects, for example a hand of the user. Asdescribed herein, the totems may not actually have any physical inputstructures (e.g., keys, triggers, joystick, trackball, rocker switch).

Instead, the totem may simply provide a physical surface, and the ARsystem may render a user interface so as to appear to a user to be onone or more surfaces of the totem. For example, and as discussed in moredetail further herein, the AR system may render an image of a computerkeyboard and trackpad to appear to reside on one or more surfaces of atotem. For instance, the AR system may render a virtual computerkeyboard and virtual trackpad to appear on a surface of a thinrectangular plate of aluminum which serves as a totem. The rectangularplate does not itself have any physical keys or trackpad or sensors.However, the AR system may detect user manipulation or interaction ortouches with the rectangular plate as selections or inputs made via thevirtual keyboard and/or virtual trackpad. Many of these components aredescribed in detail elsewhere herein.

Capturing 3D Points and Creating Passable Worlds

With a system such as that depicted in FIG. 17 and other figures above,3-D points may be captured from the environment, and the pose (i.e.,vector and/or origin position information relative to the world) of thecameras that capture those images or points may be determined, so thatthese points or images may be “tagged”, or associated, with this poseinformation. Then points captured by a second camera may be utilized todetermine the pose of the second camera. In other words, one can orientand/or localize a second camera based upon comparisons with taggedimages from a first camera.

Then this knowledge may be utilized to extract textures, make maps, andcreate a virtual copy of the real world (because then there are twocameras around that are registered). So at the base level, in oneembodiment you have a person-worn system that can be utilized to captureboth 3-D points and the 2-D images that produced the points, and thesepoints and images may be sent out to a cloud storage and processingresource. They may also be cached locally with embedded pose information(i.e., cache the tagged images); so the cloud may have on the ready(i.e., in available cache) tagged 2-D images (i.e., tagged with a 3-Dpose), along with 3-D points. If a user is observing something dynamic,he may also send additional information up to the cloud pertinent to themotion (for example, if looking at another person's face, the user cantake a texture map of the face and push that up at an optimizedfrequency even though the surrounding world is otherwise basicallystatic).

The cloud system may be configured to save some points as fiducials forpose only, to reduce overall pose tracking calculation. Generally it maybe desirable to have some outline features to be able to track majoritems in a user's environment, such as walls, a table, etc., as the usermoves around the room, and the user may want to be able to “share” theworld and have some other user walk into that room and also see thosepoints. Such useful and key points may be termed “fiducials” becausethey are fairly useful as anchoring points—they are related to featuresthat may be recognized with machine vision, and that can be extractedfrom the world consistently and repeatedly on different pieces of userhardware. Thus these fiducials preferably may be saved to the cloud forfurther use.

In one embodiment it is preferable to have a relatively evendistribution of fiducials throughout the pertinent world, because theyare the kinds of items that cameras can easily use to recognize alocation.

In one embodiment, the pertinent cloud computing configuration may beconfigured to groom the database of 3-D points and any associated metadata periodically to use the best data from various users for bothfiducial refinement and world creation. In other words, the system maybe configured to get the best dataset by using inputs from various userslooking and functioning within the pertinent world. In one embodimentthe database is intrinsically fractal—as users move closer to objects,the cloud passes higher resolution information to such users. As a usermaps an object more closely, that data is sent to the cloud, and thecloud can add new 3-D points and image-based texture maps to thedatabase if they are better than what has been previously stored in thedatabase. All of this may be configured to happen from many userssimultaneously.

As described above, an augmented or virtual reality experience may bebased upon recognizing certain types of objects. For example, it may beimportant to understand that a particular object has a depth in order torecognize and understand such object. Recognizer software objects(“recognizers”) may be deployed on cloud or local resources tospecifically assist with recognition of various objects on either orboth platforms as a user is navigating data in a world. For example, ifa system has data for a world model comprising 3-D point clouds andpose-tagged images, and there is a desk with a bunch of points on it aswell as an image of the desk, there may not be a determination that whatis being observed is, indeed, a desk as humans would know it. In otherwords, some 3-D points in space and an image from someplace off in spacethat shows most of the desk may not be enough to instantly recognizethat a desk is being observed.

To assist with this identification, a specific object recognizer may becreated that will go into the raw 3-D point cloud, segment out a set ofpoints, and, for example, extract the plane of the top surface of thedesk. Similarly, a recognizer may be created to segment out a wall from3-D points, so that a user could change wallpaper or remove part of thewall in virtual or augmented reality and have a portal to another roomthat is not actually there in the real world. Such recognizers operatewithin the data of a world model and may be thought of as software“robots” that crawl a world model and imbue that world model withsemantic information, or an ontology about what is believed to existamongst the points in space. Such recognizers or software robots may beconfigured such that their entire existence is about going around thepertinent world of data and finding things that it believes are walls,or chairs, or other items. They may be configured to tag a set of pointswith the functional equivalent of, “this set of points belongs to awall”, and may comprise a combination of point-based algorithm andpose-tagged image analysis for mutually informing the system regardingwhat is in the points.

Object recognizers may be created for many purposes of varied utility,depending upon the perspective. For example, in one embodiment, apurveyor of coffee such as Starbucks may invest in creating an accuraterecognizer of Starbucks coffee cups within pertinent worlds of data.Such a recognizer may be configured to crawl worlds of data large andsmall searching for Starbucks coffee cups, so they may be segmented outand identified to a user when operating in the pertinent nearby space(i.e., perhaps to offer the user a coffee in the Starbucks outlet rightaround the corner when the user looks at his Starbucks cup for a certainperiod of time).

With the cup segmented out, it may be recognized quickly when the usermoves it on his desk. Such recognizers may be configured to run oroperate not only on cloud computing resources and data, but also onlocal resources and data, or both cloud and local, depending uponcomputational resources available. In one embodiment, there is a globalcopy of the world model on the cloud with millions of users contributingto that global model, but for smaller worlds or sub-worlds like anoffice of a particular individual in a particular town, most of theglobal world will not care what that office looks like, so the systemmay be configured to groom data and move to local cache information thatis believed to be most locally pertinent to a given user.

In one embodiment, for example, when a user walks up to a desk, relatedinformation (such as the segmentation of a particular cup on his table)may be configured to reside only upon his local computing resources andnot on the cloud, because objects that are identified as ones that moveoften, such as cups on tables, need not burden the cloud model andtransmission burden between the cloud and local resources.

Thus the cloud computing resource may be configured to segment 3-Dpoints and images, thus factoring permanent (i.e., generally not moving)objects from movable ones, and this may affect where the associated datais to remain, where it is to be processed, remove processing burden fromthe wearable/local system for certain data that is pertinent to morepermanent objects, allow one-time processing of a location which thenmay be shared with limitless other users, allow multiple sources of datato simultaneously build a database of fixed and movable objects in aparticular physical location, and segment objects from the background tocreate object-specific fiducials and texture maps.

In one embodiment, the system may be configured to query a user forinput about the identity of certain objects (for example, the system maypresent the user with a question such as, “is that a Starbucks coffeecup?”), so that the user may train the system and allow the system toassociate semantic information with objects in the real world. Anontology may provide guidance regarding what objects segmented from theworld can do, how they behave, etc. In one embodiment the system mayfeature a virtual or actual keypad, such as a wirelessly connectedkeypad, connectivity to a keypad of a smartphone, or the like, tofacilitate certain user input to the system.

The system may be configured to share basic elements (walls, windows,desk geometry, etc.) with any user who walks into the room in virtual oraugmented reality, and in one embodiment that person's system will beconfigured to take images from his particular perspective and uploadthose to the cloud. Then the cloud becomes populated with old and newsets of data and can run optimization routines and establish fiducialsthat exist on individual objects.

GPS and other localization information may be utilized as inputs to suchprocessing. Further, other computing systems and data, such as one'sonline calendar or Facebook® account information, may be utilized asinputs (for example, in one embodiment, a cloud and/or local system maybe configured to analyze the content of a user's calendar for airlinetickets, dates, and destinations, so that over time, information may bemoved from the cloud to the user's local systems to be ready for theuser's arrival time in a given destination).

In one embodiment, tags such as QR codes and the like may be insertedinto a world for use with non-statistical pose calculation,security/access control, communication of special information, spatialmessaging, non-statistical object recognition, etc.

In one embodiment, cloud resources may be configured to pass digitalmodels of real and virtual worlds between users, as described above inreference to “passable worlds”, with the models being rendered by theindividual users based upon parameters and textures. This reducesbandwidth relative to the passage of realtime video, allows rendering ofvirtual viewpoints of a scene, and allows millions or more users toparticipate in one virtual gathering without sending each of them datathat they need to see (such as video), because their views are renderedby their local computing resources.

The virtual reality system (“VRS”) may be configured to register theuser location and field of view (together known as the “pose”) throughone or more of the following: realtime metric computer vision using thecameras, simultaneous localization and mapping techniques, maps, anddata from sensors such as gyros, accelerometers, compass, barometer,GPS, radio signal strength triangulation, signal time of flightanalysis, LIDAR ranging, RADAR ranging, odometry, and sonar ranging. Thewearable device system may be configured to simultaneously map andorient. For example, in unknown environments, the VRS may be configuredto collect information about the environment, ascertaining fiducialpoints suitable for user pose calculations, other points for worldmodeling, images for providing texture maps of the world. Fiducialpoints may be used to optically calculate pose.

As the world is mapped with greater detail, more objects may besegmented out and given their own texture maps, but the world stillpreferably is representable at low spatial resolution in simple polygonswith low resolution texture maps. Other sensors, such as those discussedabove, may be utilized to support this modeling effort. The world may beintrinsically fractal in that moving or otherwise seeking a better view(through viewpoints, “supervision” modes, zooming, etc.) requesthigh-resolution information from the cloud resources. Moving closer toobjects captures higher resolution data, and this may be sent to thecloud, which may calculate and/or insert the new data at interstitialsites in the world model.

Referring to FIG. 18, the wearable AR system may be configured tocapture image information and extract fiducials and recognized points(52). The wearable local system may calculate pose using one of the posecalculation techniques discussed below. The cloud (54) may be configuredto use images and fiducials to segment 3-D objects from more static 3-Dbackground; images provide textures maps for objects and the world(textures may be realtime videos). The cloud resources (56) may beconfigured to store and make available static fiducials and textures forworld registration. The cloud resources may be configured to groom thepoint cloud for optimal point density for registration.

The cloud resources (60) may store and make available object fiducialsand textures for object registration and manipulation; the cloud maygroom point clouds for optimal density for registration. The couldresource may be configured (62) to use all valid points and textures togenerate fractal solid models of objects; the cloud may groom pointcloud information for optimal fiducial density. The cloud resource (64)may be configured to query users for training on identity of segmentedobjects and the world; an ontology database may use the answers to imbueobjects and the world with actionable properties.

The passable world model essentially allows a user to effectively passover a piece of the user's world (i.e., ambient surroundings,interactions, etc.) to another user. Each user's respective individualAR system (e.g., Sensorywear™ augmented reality devices) capturesinformation as the user passes through or inhabits an environment, whichthe AR system processes to produce a passable world model. Theindividual AR system may communicate or pass the passable world model toa common or shared collection of data, referred to as the cloud. Theindividual AR system may communicate or pass the passable world model toother users, either directly or via the cloud. The passable world modelprovides the ability to efficiently communicate or pass information thatessentially encompasses at least a field of view of a user. In oneembodiment, the system uses the pose and orientation information, aswell as collected 3D points described above in order to create thepassable world.

Referring now to FIG. 19, similar to the system described in FIG. 17,the passable world system comprises one or more user AR systems or userdevices 208 (e.g., 208 a, 208 b, 208 c) that are able to connect to thecloud network 204, a passable world model 202, a set of objectrecognizers 210, and a database 206. The cloud server may be a LAN, aWAN or any other network.

As shown in FIG. 19, the passable world model is configured to receiveinformation from the user devices 208 and also transmit data to themthrough the network. For example, based on the input from a user, apiece of the passable world may be passed on from one user to the other.The passable world model may be thought of collection of images, pointsand other information based on which the AR system is able to construct,update and build the virtual world on the cloud, and effectively passpieces of the virtual world to various users.

For example, a set of points collects from user device 208 may becollected in the passable world model 202. Various object recognizers210 may crawl through the passable world model 202 to recognize objects,tag images, etc., and attach semantic information to the objects, aswill be described in further detail below. The passable world model 202may use the database 206 to build its knowledge of the world, attachsemantic information, and store data associated with the passable world.

FIG. 20 illustrates aspects of a passable world model 4020 according toone illustrated embodiment. As a user walks through an environment, theuser's individual AR system captures information (e.g., images) andsaves the information posed tagged images, which form the core of thepassable world model, as shown by multiple keyframes (cameras) that havecaptured information about the environment. The passable world model isa combination of raster imagery, point+descriptors clouds, andpolygonal/geometric definitions (referred to herein as parametricgeometry).

All this information is uploaded to and retrieved from the cloud, asection of which corresponds to this particular space that the user haswalked into. As shown in FIG. 19, the passable world model also containsmany object recognizers that work on the cloud (or on the user'sindividual system) to recognize objects in the environment based onpoints and pose-tagged images captured through the various keyframes ofmultiple users.

Asynchronous communications is established between the user's respectiveindividual AR system and the cloud based computers (e.g., servercomputers). In other words, the user's individual AR system (e.g.,user's sensorywear) is constantly updating information about the user'ssurroundings to the cloud, and also receiving information from the cloudabout the passable world. Thus, rather than each user having to captureimages, recognize objects of the images etc., having an asynchronoussystem allows the system to be more efficient. Information that alreadyexists about that part of the world is automatically communicated to theindividual AR system while new information is updated to the cloud. Itshould be appreciated that the passable world model lives both on thecloud or other form of networking computing or peer to peer system, andalso may live on the user's individual system.

The AR system may employ different levels of resolutions for the localcomponents (e.g., computational component such as belt pack) and remotecomponents (e.g., cloud based computers) which are typically morecomputationally powerful than local components. The cloud basedcomputers may pick data collected by the many different individual ARsystems, and optionally from one or more space or room based sensorsystems. The cloud based computers may aggregate only the best (i.e.,most useful) information into a persistent world model.

FIG. 21 illustrates an exemplary method 2100 of interacting with thepassable world model. First, the user's individual AR system may detecta location of the user (step 2102). The location may be derived by thetopological map of the system, as will be described in further detailbelow. The location may be derived by GPS or any other localizationtool. It should be appreciated that the passable world is constantlyaccessed by the individual system.

In another embodiment (not shown), the user may request access toanother user's space, prompting the system to access the section of thepassable world, and associated parametric information corresponding tothe other user. Thus, there may be many triggers for the passable world.At the simplest level, however, it should be appreciated that thepassable world is constantly being updated and accessed by multiple usersystems, thereby constantly adding and receiving information from thecloud.

Following the above example, based on the known location of the user,the system may draw a radius denoting a physical area around the userthat communicates both the position and intended direction of the user(step 2104). Next, the system may retrieve the piece of the passableworld based on the anticipated position of the user (step 2106) Next,the system may upload information obtained from the user's environmentto the passable world mode (step 2108) and render the passable worldmodel associated with the position of the user (step 2110).

The piece of the passable world may contain information from thegeometric map of the space acquired through previous keyframes andcaptured images and data that is stored in the cloud. Having thisinformation enables virtual content to meaningfully interact with theuser's real surroundings in a coherent manner. For example, the user maywant to leave a virtual object for a friend in a real space such thatthe friend, when he/she enters the real space finds the virtual object.Thus, it is important for the system to constantly access the passableworld to retrieve and upload information. It should be appreciated thatthe passable world contains a persistent digital representations of realspaces that is important in rendering virtual or digital content inrelation to real coordinates of a physical space.

It should be appreciated that the passable world model does not itselfrender content that is displayed to the user. Rather it is a high levelconcept of dynamically retrieving and updating a persistent digitalrepresentation of the real world in the cloud. The derived geometricinformation is loaded onto a game engine, which actually does therendering of the content associated with the passable world.

Thus, regardless of whether the user is in a particular space or not,that particular space has a digital representation in the cloud that canbe accessed by any user. This piece of the passable world may containinformation about the physical geometry of the space and imagery of thespace, information about various avatars that are occupying the space,information about virtual objects and other miscellaneous information.

As described in detail further herein, object recognizers, examine or“crawl” the passable world models, tagging points that belong toparametric geometry. Parametric geometry and points+descriptors arepackaged as passable world models, to allow low latency passing orcommunicating of information which defines a portion of a physical worldor environment. The AR system can implement a two tier structure, inwhich the passable world model allow fast pose in a first tier, but theninside that framework a second tier (e.g., FAST® features) can increaseresolution by performing a frame-to-frame based three-dimensional (3D)feature mapping, than tracking.

FIG. 22 illustrates an exemplary method 2200 of recognizing objectsthrough object recognizers. When a user walks into a room, the user'ssensorywear captures information (e.g., pose tagged images) about theuser's surroundings from multiple points of view (step 2202). Forexample, by the time the user walks into a section of a room, the user'sindividual AR system has already captured numerous keyframes and posetagged images about the surroundings. It should be appreciated that eachkeyframe may include information about the depth and color of theobjects in the surroundings. Next, the object recognizer extracts a setof sparse 3D points from the images (step 2204).

Next, the object recognizer (either locally or in the cloud) uses imagesegmentation to find a particular object in the keyframe (step 2206). Itshould be appreciated that different objects have different objectrecognizers that have been written and programmed to recognize thatparticular object. For illustrative purposes, the following example,will assume that the object recognizer recognizes doors.

The object recognizer may be an autonomous and atomic software object“robot” that takes pose tagged images of the space, key frames, 2D or 3Dfeature points, and geometry of the space to recognize the door. Itshould be appreciated that multiple object recognizers may runsimultaneously on a set of data, and they can run independent of eachother. It should be appreciated that the object recognizer takes 2Dimages of the object (2D color information, etc.), 3D images (depthinformation) and also takes 3D sparse points to recognize the object ina geometric coordinate frame of the world.

Next, the object recognizer may correlate the 2D segmented imagefeatures with the sparse 3D points to derive, using 2D/3D data fusion,object structure and properties. For example, the object recognizer mayidentify specific geometry of the door with respect the key frames.Next, based on this, the object recognizer parameterizes the geometry ofthe object (step 2208). For example, the object recognizer may attachsemantic information to the geometric primitive (e.g., the door has ahinge, the door can rotate 90 degrees, etc.). Or, the object recognizermay reduce the size of the door, etc. Next, the object recognizer maysynchronize the parametric geometry to the cloud (step 2210).

Next, after recognition, the object recognizer re-inserts the geometricand parametric information into the passable world model (step 2212).For example, the object recognizer may dynamically estimate the angle ofthe door, and insert it into the world. Thus, it can be appreciated thatusing the object recognizer allows the system to save computationalpower because rather than constant real-time capture of informationabout the angle of the door or movement of the door, the objectrecognizer uses the stored parametric information to estimate themovement or angle of the door. This information may be updated to thecloud so that other users can see the angle of the door in variousrepresentations of the passable world.

As briefly discussed above, object recognizers are atomic autonomoussoftware and/or hardware modules which ingest sparse points (i.e., notnecessarily a dense point cloud), pose-tagged images, and geometry, andproduce parametric geometry that has semantics attached. The semanticsmay take the form of taxonomical descriptor, for example “wall,”“chair,” “Aeron® chair,” and properties or characteristics associatedwith the taxonomical descriptor.

For example, a taxonomical descriptor such as a table may haveassociated descriptions such as “has a flat horizontal surface which cansupport other objects.” Given an ontology, an object recognizer turnsimages, points, and optionally other geometry, into geometry that hasmeaning (i.e., semantics).

Since the individual AR systems are intended to operate in the realworld environment, the points represent sparse, statistically relevant,natural features. Natural features are those that are inherent to theobject (e.g., edges, holes), in contrast to artificial features added(e.g., printed, inscribed or labeled) to objects for the purpose ofmachine-vision recognition. The points do not necessarily need to bevisible to humans. The points are not limited to point features, e.g.,line features and high dimensional features.

Object recognizers may be categorized into two types, Type 1—BasicObjects (e.g., walls, cups, chairs, etc.), Type 2—Detailed Objects(e.g., Aeron® chair, my wall). In some implementations, the Type 1recognizers run across the entire cloud, while the Type 2 recognizersrun against previously found Type 1 data (e.g., search all chairs forAeron® chairs). The object recognizers may use inherent properties of anobject to facilitate in object identification. Or, the objectrecognizers may use ontological relationship between objects tofacilitate implementation. For example, an object recognizer may use thefact that window must be in a wall to facilitate recognition ofinstances of windows.

Object recognizers will typically be bundled, partnered or logicallyassociated with one or more applications. For example, a cup finderobject recognizer may be associated with one, two or more applicationsin which identifying a presence of a cup in a physical space would beuseful. Applications can be logically connected for associated withdefined recognizable visual data or models. For example, in response toa detection of any Aeron® chairs in an image, the AR system calls orexecutes an application from the Herman Miller Company, the manufacturerand/or seller of Aeron® chairs. Similarly, in response to detection of aStarbucks® signs or logo in an image, the AR system calls or executes aStarbucks® application.

As an example, the AR system may employ an instance of a generic wallfinder object recognizer. The generic wall finder object recognizeridentifies instances of walls in image information, without regard tospecifics about a wall. Thus, the generic wall finder object recognizeridentifies vertically oriented surfaces that constitute walls in theimage data. The AR system may also employ an instance of a specific wallfinder object recognizer, which is separate and distinct from thegeneric wall finder. The specific wall finder object recognizeridentifies vertically oriented surfaces that constitute walls in theimage data and which have one or more specific characteristics beyondthose of generic wall.

For example, a given specific wall may have one or more windows indefined positions, one or more doors in defined positions, may have adefined paint color, may have artwork hung from the wall, etc., whichvisually distinguishes the specific wall from other walls. Such allowsthe specific wall finder object recognizer to identify particular walls.For example, one instance of a specific wall finder object recognizermay identify a wall of a user's office. Other instances of specific wallfinder object recognizers may identify respective walls of a user'sliving room or bedroom.

A specific object recognizer may stand independently from a genericobject recognizer. For example, a specific wall finder object recognizermay run completely independently from a generic wall finder objectrecognizer, not employing any information produced by the generic wallfinder object recognizer. Alternatively, a specific (i.e., more refined)object recognizer may be run nested against objects previously found bya more generic object recognizer. For example, a generic and/or aspecific door finder object recognizer may run against a wall found by ageneric and/or specific wall finder object recognizer, since a door mustbe in a wall. Likewise, a generic and/or a specific window finder objectrecognizer may run against a wall found by a generic and/or specificwall finder object recognizer, since a window must be in a wall.

An object recognizer may not only identify the existence or presences ofan object, but may identify other characteristics associated with theobject. For example, a generic or specific door finder object recognizermay identify a type of door, whether the door is hinged or sliding,where the hinge or slide is located, whether the door is currently in anopen or a closed position, and/or whether the door is transparent oropaque, etc.

As noted above, each object recognizer is atomic, that is they areautonomic, autonomous, asynchronous, essentially a black box softwareobject. This allows object recognizers to be community built. Thebuilding of object recognizers may be incentivized with variousincentives. For example, an online marketplace or collection point forobject recognizers may be established. Object recognizer developers maybe allowed of post object recognizers for linking or associating withapplications developed by other object recognizer or applicationdevelopers.

Various incentives may be provided. For example, an incentive may beprovided for posting of an object recognizer. Also for example, anincentive may be provided to an object recognizer developer or authorbased on the number of times an object recognizer is logicallyassociated with an application and/or based on the total number ofdistributions of an application to which the object recognizer islogically associated. As a further example, an incentive may be providedto an object recognizer developer or author based on the number of timesan object recognizer is used by applications that are logicallyassociated with the object recognizer. The incentives may be monetaryincentives, may provide access to services or media behind a pay wall,and/or credits for acquiring services, media, or goods.

It would, for example, be possible to instantiate 10,000 or moredistinct generic and/or specific object recognizers. These genericand/or specific object recognizers can all be run against the same data.As noted above, some object recognizers can be nested, essentiallylayered on top of each other.

A control program may control the selection, use or operation of thevarious object recognizers, for example arbitrating the use or operationthereof. Some object recognizers may be placed in different regions, toensure that the object recognizers do not overlap each other. One, moreor even all of the object recognizers can run locally at the user, forexample on the computation component (e.g., belt pack). One, more oreven all of the object recognizers can run remotely from the user, forexample on the cloud server computers.

Object recognizers are related to Apps in the ecosystem. Eachapplication has an associated list of object recognizers it requires.Extensible, can write own apps and recognizers. Could run locally onbelt pack, or submit to app store. Monetize apps and object recognizers,e.g., small royalty to author for each download and/or each successfuluse of object recognizer.

In some implementations, a user may train an AR system, for examplemoving through a desired set of movements. In response, the AR systemmay generate an avatar sequence in which an avatar replicates themovements, for example animating the avatar. Thus, the AR systemcaptures or receives images of a user, and generates animation of anavatar based on movements of the user in the captured images. The usermay be instrumented, for example wearing one or more sensors. The ARsystem knows where the pose of the user's head, eyes, and/or hands. Theuser can, for example, simply act out some motions they want to train.The AR system preforms a reverse kinematics analysis of the rest ofuser's body, and makes an animation based on the reverse kinematicsanalysis.

Avatars in the Passable World

The passable world also contains information about various avatarsinhabiting a space. It should be appreciated that every user may berendered as an avatar in one embodiment. Or, a user operatingsensorywear from a remote location can create an avatar and digitallyoccupy a particular space as well.

In either case, since the passable world is not a static data structure,but rather constantly receives information, avatar rendering and remotepresence of users into a space may be based on the user's interactionwith the user's individual AR system. Thus, rather than constantlyupdating an avatar's movement based on captured keyframes, as capturedby cameras, avatars may be rendered based on a user's interaction withhis/her sensorywear device.

More particularly, the user's individual AR system contains informationabout the user's head pose and orientation in a space, information abouthand movement etc. of the user, information about the user's eyes andeye gaze, information about any totems that are being used by the user.Thus, the user's individual AR system already holds a lot of informationabout the user's interaction within a particular space that istransmitted to the passable world model. This information may then bereliably used to create avatars for the user and help the avatarcommunicate with other avatars or users of that space. It should beappreciated that no third party cameras are needed to animate theavatar, rather, the avatar is animated based on the user's individual ARsystem.

For example, if the user is not in currently at a conference room, butwants to insert an avatar into that space to participate in a meeting atthe conference room, the AR system takes information about the user'sinteraction with his/her own system and uses those inputs to render theavatar into the conference room through the passable world model.

The avatar may be rendered such that the avatar takes the form of theuser's own image such that it looks like the user himself/herself isparticipating in the conference. Or, based on the user's preference, theavatar may be any image chosen by the user. For example, the user mayrender himself/herself as a bird that flies around the space of theconference room.

At the same time, information about the conference room (e.g., keyframes, points, pose-tagged images, avatar information of people in theconference room, recognized objects, etc.) are rendered to the user whois not currently in the conference room. In the physical space, thesystem may have captured keyframes that are geometrically registered andderives points from the keyframes.

As discussed above, based on these points, the system calculates poseand runs object recognizers, and reinserts parametric geometry into thekeyframes, such that the points of the keyframes also have semanticinformation attached to them. Thus, with all this geometric and semanticinformation, the conference room may now be shared with other users. Forexample, the conference room scene may be rendered on the user's table.Thus, even if there is no camera at the conference room, the passableworld model, using information collected through prior key frames etc.,is able to transmit information about the conference room to other usersand recreate the geometry of the room for other users in other spaces.

Topological Map

It should be appreciated that the AR system may use topological maps forlocalization purposes rather than using geometric maps created fromextracted points and pose tagged images. The topological map is asimplified representation of physical spaces in the real world that iseasily accessible from the cloud and only presents a fingerprint of aspace, and the relationship between various spaces.

The AR system may layer topological maps on the passable world model,for example to localize nodes. The topological map can layer varioustypes of information on the passable world model, for instance: pointcloud, images, objects in space, global positioning system (GPS) data,Wi-Fi data, histograms (e.g., color histograms of a room), receivedsignal strength (RSS) data, etc.

In order to create a complete virtual world that maybe reliably passedbetween various users, the AR system captures information (e.g., mappoints, features, pose tagged images, objects in a scene, etc.) that isstored in the cloud, and then retrieved as needed. As discussedpreviously, the passable world model is a combination of raster imagery,point+descriptors clouds, and polygonal/geometric definitions (referredto herein as parametric geometry). Thus, it should be appreciated thatthe sheer amount of information captured through the users' individualAR system allows for high quality and accuracy in creating the virtualworld. However, for localization purposes, sorting through that muchinformation to find the piece of passable world most relevant to theuser is highly inefficient and costs bandwidth.

To this end, the AR system creates a topological map that essentiallyprovides less granular information about a particular scene or aparticular place. The topological map may be derived through globalpositioning system (GPS) data, Wi-Fi data, histograms (e.g., colorhistograms of a room), received signal strength (RSS) data, etc. Forexample, the topological map may use a color histogram of a particularroom, and use it as a node in the topological map. In doing so, the roomhas a distinct signature that is different from any other room or place.

Thus, although the histogram will not contain particular informationabout all the features and points that have been captured by variouscameras (keyframes), the system may immediately detect, based on thehistogram, where the user is, and then retrieve all the more particulargeometric information associated with that particular room or place.Thus, rather than sorting through the vast amount of geometric andparametric information that encompasses that passable world model, thetopological map allows for a quick and efficient way to localize, andthen only retrieve the keyframes and points most relevant to thatlocation.

For example, after the system has determined that the user is in aconference room of a building, the system may then retrieve all thekeyframes and points associated with the conference room rather thansearching through all the geometric information stored in the cloud.

For example, the AR system can represent two images captured byrespective cameras of a part of the same scene in a graph theoreticcontext as first and second pose tagged images. It should be appreciatedthat the cameras in this context may refer to a single camera takingimages of different scenes, or it may be two cameras. There is somestrength of connection between the pose tagged images, which could forexample be the points that are in the field of views of both of thecameras. The cloud based computer constructs such as a graph (i.e., atopological representation of a geometric world). The total number ofnodes and edges in the graph is much smaller than the total number ofpoints in the images.

At a higher level of abstraction higher, other information monitored bythe AR system can be hashed together. For example, the cloud basedcomputer(s) may hash together one or more of global positioning system(GPS) location information, Wi-Fi location information (e.g., signalstrengths), color histograms of a physical space, and/or informationabout physical objects around a user. The more points of data, the morelikely that the computer will statistically have a unique identifier forthat space. In this case, space is a statistically defined concept. Forexample, in a graph each node may have a histogram profile.

As an example, an office may be a space that is represented as, forexample 500 points and two dozen pose tagged images. The same space maybe represented topologically as a graph having only 25 nodes, and whichcan be easily hashed against. Graph theory allows representation ofconnectedness, for example as a shortest path algorithmically betweentwo spaces.

Thus, the system abstracts away from the specific geometry by turningthe geometry into pose tagged images having implicit topology. Thesystem takes the abstraction a level higher by adding other pieces ofinformation, for example color histogram profiles, and the Wi-Fi signalstrengths. This makes it easier for the system to identify an actualreal world location of a user without having to understand or processall of the geometry associated with the location.

Referring now to FIG. 23, the topological map 2300, in one embodiment,may simply be a collection of nodes and lines. Each node may represent aparticular localized location (e.g., the conference room of an officebuilding) having a distinct signature (e.g., GPS information, histogram,Wi-Fi data, RSS data etc.) and the lines may represent the connectivitybetween them. It should be appreciated that the connectivity may nothave anything to do with geographical connectivity, but rather may be ashared device or a shared user. Thus, layering the topological map onthe geometric map is especially helpful for localization and efficientlyretrieving only relevant information from the cloud.

FIG. 24 illustrates an exemplary method 2400 of constructing atopological map. First, the user's individual AR system may take a wideangle camera picture of a particular location (step 2402), andautomatically generate a color histogram of the particular location(step 2406). As discussed above, the system may use any other type ofidentifying information, (Wi-Fi data, RSS information, GPS data, numberof windows, etc.) but the color histogram is used in this example forillustrative purposes.

Next, the system runs a search to identify the location of the user bycomparing the color histogram to a database of color histograms storedin the cloud. (step 2408) Next, the system determines if the colorhistogram matches an existing histogram (step 2410). If the colorhistogram does not match any color histogram of the database of colorhistograms, it may then be stored in the cloud. Next, the particularlocation having the distinct color histogram is stored as a node in thetopological map (step 2414).

Next, the user may walk into another location, where the user'sindividual AR system takes another picture and generates another colorhistogram of the other location. If the color histogram is the same asthe previous color histogram or any other color histogram, the AR systemidentifies the location of the user (step 2412). Here, since the firstnode and second node were taken by the same user (or same camera/sameindividual user system), the two nodes are connected in the topologicalmap.

In addition to localization, the topological map may also be used tofind loop-closure stresses in geometric maps or geometric configurationsof a particular place. It should be appreciated that for any givenspace, images taken by the user's individual AR system (multiple fieldof view images captured by one user's individual AR system or multipleusers' AR systems) give rise a large number of map points of theparticular space.

For example, a single room may have a thousand map points capturedthrough multiple points of views of various cameras (or one cameramoving to various positions). Thus, if a camera (or cameras) associatedwith the users' individual AR system captures multiple images, a largenumber of points are collected and transmitted to the cloud. Thesepoints not only help the system recognize objects, as discussed above,and create a more complete virtual world that may be retrieved as partof the passable world model, they also enable refinement of calculationof the position of the camera based on the position of the points. Inother words, the collected points may be used to estimate the pose(e.g., position and orientation) of the keyframe (e.g. camera) capturingthe image.

It should be appreciated, however, that given the large number of mappoints and keyframes, there are bound to be some errors (i.e., stresses)in this calculation of keyframe position based on the map points. Toaccount for these stresses, the AR system may perform a bundle adjust. Abundle adjust allows for the refinement, or optimization of the mappoints and keyframes to minimize the stresses in the geometric map.

For example, as illustrated in FIG. 25, the geometric map 2500 may be acollection of keyframes that are all connected to each other. Forexample, each node of the geometric map may represent a keyframe. Thestrength of lines between the keyframes may represent the number offeatures or map points shared between them. For example, if a firstkeyframe and a second keyframe are close together, they may share alarge number of map points, and may thus be represented with a thickerconnecting line.

It should be appreciated that other ways of representing geometric mapsmay be similarly used. For example, the strength of the line may bebased on a geographical proximity, in another embodiment. Thus, as shownin FIG. 25, each geometric map may represent a large number of keyframesand their connection to each other. Now, assuming that a stress isidentified in a particular point of the geometric map, by performing abundle adjust, the stress may be alleviated by radially pushing thestress out from the particular point in waves propagating from theparticular point of stress.

The following paragraph illustrates an exemplary method of performing awave propagation bundle adjust. It should be appreciated that all theexamples below refer solely to wave propagation bundle adjusts. First, aparticular point of stress is identified. For example, the system maydetermine that the stress at a particular point of the geometric map isespecially high (e.g., residual errors, etc.).

The stress may be identified based on one of two reasons. One, a maximumresidual error may be defined for the geometric map. If a residual errorat a particular point is greater than the predefined maximum residualerror, a bundle adjust may be initiation. Second, a bundle adjust may beinitiated in the case of loop closures, as will be described furtherbelow (when a topological map indicates that mis-alignments of mappoints)

Next, the system distributes the error evenly starting with the point ofstress and propagating it radially through a network of nodes thatsurround the particular point of stress. For example, referring back toFIG. 25, the bundle adjust may distribute the error to n=1 around theidentified point of stress.

Next, the system may propagate the stress even further, and push out thestress to n=2, or n=3 such that the stress is radially pushed outfurther and further until the stress is distributed evenly. Thus,performing the bundle adjust is an important way of reducing stress inthe geometric maps, and helps optimize the points and keyframes.Ideally, the stress is pushed out to n=2 or n=3 for better results.

It should be appreciated, that the waves may be propagated in smallerincrements. For example, after the wave has been pushed out to n=2around the point of stress, a bundle adjust can be performed in the areabetween n=3 and n=2, and propagated radially. Thus, this iterative wavepropagating bundle adjust process can be run on massive data.

In an optional embodiment, because each wave is unique, the nodes thathave been touched by the wave (i.e., bundle adjusted) may be colored sothat the wave does not re-propagate on an adjusted section of thegeometric map. In another embodiment, nodes may be colored so thatsimultaneous waves may propagate/originate from different points in thegeometric map.

As discussed previously, layering the topological map on the geometricmap of keyframes and map points may be especially crucial in findingloop-closure stresses. A loop-closure stress refers to discrepanciesbetween map points captured at different times that should be alignedbut are mis-aligned. For example, if a user walks around the block andreturns to the same place, map points derived from the position of thefirst keyframe and the map points derived from the position of the lastkeyframe as extrapolated from the collected map points should ideally beidentical.

However, given stresses inherent in the calculation of pose (position ofkeyframes) based on the map points, there are often errors and thesystem does not recognize that the user has come back to the sameposition because estimated key points from the first key frame are notgeometrically aligned with map points derived from the last keyframe.This may be an example of a loop-closure stress.

To this end, the topological map may be used to find the loop-closurestresses. Referring back to the previous example, using the topologicalmap along with the geometric map allows the system to recognize theloop-closure stress in the geometric map because the topological map mayindicate that the user is back to the starting point (based on the colorhistogram, for example). For, example, referring to FIG. 26, plot 2600shows that the color histogram of keyframe B, based on the topologicalmap may be the same as keyframe A. Based on this, the system detectsthat A and B should be closer together in the same node, and the systemmay then perform a bundle adjust.

Thus, having identified the loop-closure stress, the system may thenperform a bundle adjust on the keyframes and map points derived fromthem that share a common topological map node. However, doing this usingthe topological map ensures that the system only retrieves the keyframeson which the bundle adjust needs to be performed instead of retrievingall the keyframes in the system. For example, if the system identifies,based on the topological map that there is a loop closure stress, thesystem may simply retrieve the keyframes associated with that particularnode of the topological map, and perform the bundle adjust on only thoseset of keyframes rather than all the keyframes of the geometric map.

FIG. 27 illustrates an exemplary algorithm 2700 for correcting loopclosure stresses based on the topological map. First, the system mayidentify a loop closure stress based on the topological map that islayered on top of the geometric map (step 2702). Once the loop closurestress has been identified, the system may retrieve the set of keyframes associated with the node of the topological map at which the loopclosure stress has occurred (step 2704). After having retrieved the keyframes of that node of the topological map, the system may initiate abundle adjust (step 2706) on that point in the geometric map, andresolves look closure stress in waves, thus propagating the errorradially away from the point of stress (step 2708).

Mapping

The AR system may employ various mapping related techniques in order toachieve high depth of field in the rendered light fields. In mapping outthe virtual world, it is important to know all the features and pointsin the real world to accurately portray virtual objects in relation tothe real world. To this end, as discussed previously, field of viewimages captured from users of the AR system are constantly adding to thepassable world model by adding in new pictures that convey informationabout various points and features of the real world.

Based on the points and features, as discussed before, one can alsoextrapolate the pose and position of the keyframe (e.g., camera, etc.).While this allows the AR system to collect a set of features (2D points)and map points (3D points), it may also be important to find newfeatures and map points to render a more accurate version of thepassable world.

One way of finding new map points and/or features may be to comparefeatures of one image against another. Each feature may have a label orfeature descriptor attached to it (e.g., color, identifier, etc.).Comparing the labels of features in one picture to another picture maybe one way of uniquely identifying natural features in the environment.For example, if there are two keyframes, each of which captures about500 features, comparing the features of one keyframe with another mayhelp determine if there are new points. However, while this might be afeasible solution when there are just two keyframes, it becomes a verylarge search problem that takes up a lot of processing power when thereare multiple keyframes, each having many points. In other words, ifthere are M keyframes, each having N unmatched features, searching fornew features involves an operation of MN2 (O(MN2)), which is a hugesearch operation.

Thus, to avoid such a large search operation, the AR system may find newpoints by render rather than search. In other words, assuming theposition of M keyframes are known and each of them has N points, the ARsystem may project lines (or cones) from N features to the M keyframes.Referring now to FIG. 28, in this particular example, there are 6keyframes, and lines or rays are rendered (using a graphics card) fromthe 6 keyframes to the various features.

As can be seen in plot 2800 of FIG. 28 based on the intersection of therendered lines, new map points may be found. In other words, when tworendered lines intersect, the pixel coordinate of that particular mappoint in a 3D space may be 2 instead of 1 or 0. Thus, the higher theintersection of the lines at a particular point, the higher thelikelihood that there is a map point corresponding to a particularfeature in the 3D space. Thus, the intersection of rendered lines may beused to find new map points in a 3D space.

It should be appreciated that for optimization purposes, rather thanrendering lines from the keyframes, triangular cones may instead berendered from the keyframe for more accurate results. The Nth featuremay be bisector of the cone, and the half angles to the two side edgesmay be defined by the camera's pixel pitch, which runs through the lensmapping function on either side of the Nth feature. The interior of thecone may be shaded such that the bisector is the brightest and the edgeson either side of the Nth feature may be set of 0.

The camera buffer may be a summing buffer, such that bright spots mayrepresent candidate locations of new features, but taking into accountboth camera resolution and lens calibration. In other words, projectingcones, rather than lines may help compensate for the fact that certainkeyframes are farther away than others that may have captured thefeatures at a closer distance. Thus, a cone rendered from a keyframethat is farther away will be larger (and have a large radius) than onethat is rendered from a keyframe that is closer.

It should be appreciated that for optimization purposes, triangles maybe rendered from the keyframes instead of lines. Rather than renderingsimple rays, render a triangle that is normal to the virtual camera. Asdiscussed previously, the bisector of the triangle is defined by the Nthfeature, and the half angles of the two side edges may be defined by thecamera's pixel pitch and run through a lens mapping function on eitherside of the Nth feature. Next the AR system may apply a summing bufferof the camera buffer such that the bright spots represent a candidatelocation of the features.

Essentially, the AR system may project rays or cones from a number of Nunmatched features in a number M prior key frames into a texture of theM+1 keyframe, encoding the keyframe identifier and feature identifier.The AR system may build another texture from the features in the currentkeyframe, and mask the first texture with the second. All of the colorsare a candidate pairing to search for constraints. This approachadvantageously turns the O(MN2) search for constraints into an O(MN)render, followed by a tiny O((<M)N(<<N)) search.

As a further example, the AR system may pick new keyframes based onnormals. In other words, the virtual key frame from which to view themap points may be selected by the AR system. For instance, the AR systemmay use the above keyframe projection, but pick the new “keyframe” basedon a PCA (Principal component analysis) of the normals of the Mkeyframes from which {M,N} labels are sought (e.g., the PCA-derivedkeyframe will give the optimal view from which to derive the labels).

Performing a PCA on the existing M keyframes provides a new keyframethat is most orthogonal to the existing M keyframes. Thus, positioning avirtual key frame at the most orthogonal direction may provide the bestviewpoint from which to find new map points in the 3D space. Performinganother PCA provides a next most orthogonal direction, and performing ayet another PCA provides yet another orthogonal direction. Thus, it canbe appreciated that performing 3 PCAs may provide an x, y and zcoordinates in the 3D space from which to construct map points based onthe existing M key frames having the N features.

FIG. 29 illustrates an exemplary algorithm 2900 for finding map pointsfrom M known keyframes, with no prior known map points. First, the ARsystem retrieves M keyframes associated with a particular space (step2902). As discussed previously, M keyframes refers to known keyframesthat have captured the particular space. Next, a PCA of the normal ofthe keyframes is performed to find the most orthogonal direction of theM key frames (step 2904). It should be appreciated that the PCA mayproduce three principals each of which is orthogonal to the M keyframes. Next, the AR system selects the principal that is smallest inthe 3D space, and is also the most orthogonal to the view of all thekeyframes (step 2906).

After having identified the principal that is orthogonal to thekeyframes, the AR system may place a virtual camera on the axis of theselected principal (step 2908). It should be appreciated that thevirtual keyframe may be places far away enough so that its field of viewincludes all the M keyframes.

Next, the AR system may render a feature buffer (step 2910), such that Nrays (or cones) are rendered from each of the M key frames to the Nthfeature. The feature buffer may be a summing buffer, such that thebright spots (pixel coordinates at which lines N lines have intersected)represent candidate locations of N features. It should be appreciatedthat the same process described above may be repeated with all three PCAaxes, such that map points are found on x, y and z axes.

Next, the system may store all the bright spots in the image as virtual“features” (step 2912). Next, the AR system may create a second “label”buffer at the virtual keyframe to stack the lines (or cones) and savingtheir {M, N} labels (step 2914). Next, the AR system may draw a “maskradius” around each bright spot in the feature buffer (step 2916). Themask radius represents the angular pixel error of the virtual camera.Next, the AR system may fill the circles and mask the label buffer withthe resulting binary image. It should be appreciated that in an optionalembodiment, the filling of the above circles may be bright at thecenter, fading to zero at the circumference.

In the now-masked label buffer, the AR system may, at each maskedregion, collect the principal rays using the {M, N}-tuple label of eachtriangle. It should be appreciated that if cones/triangles are usedinstead of rays, the AR system may only collect triangles where bothsides of the triangle are captured inside the circle. Thus, the maskradius essentially acts as a filter that eliminates poorly conditionedrays or rays that have a large divergence (e.g., a ray that is at theedge of a field of view (FOV) or a ray that emanates from far away).

For optimization purposes, the label buffer may be rendered with thesame shading as used previously in generated cones/triangles). Inanother optional optimization embodiment, the AR system may scale thetriangle density from one to zero instead of checking the extents(sides) of the triangles. Thus, very divergent rays will effectivelyraise the noise floor inside a masked region. Running a local thresholddetect inside the mark will trivially pull out the centroid from onlythose rays that are fully inside the mark.

Next, the AR system may feed the collection of masked/optimized rays toa bundle adjuster to estimate the location of map points (step 2918). Itshould be appreciated that this system is functionally limited to thesize of the render buffers that are employed. For example, if the keyframes are widely separated, the resulting rays/cones will have lowerresolution.

In an alternate embodiment, rather than using PCA to find the orthogonaldirection, the virtual key frame may be placed at the location of one ofthe M key frames. This may be a simpler and effective solution becausethe M key frame may have already captured the space at the bestresolution of the camera. If PCAs are used to find the orthogonaldirections at which to place the virtual keyframes, the process above isrepeated by placing the virtual camera along each PCA axis and findingmap points in each of the axes.

In yet another exemplary algorithm of finding new map points, the ARsystem may hypothesize map points. Thus, instead of using a labelbuffer, the AR system hypothesizes map points, for example by performingthe following algorithm. The AR system may first get M key frames. Next,the AR system gets the first three principal components from a PCAanalysis. Next, the AR system may place a virtual keyframe at eachprincipal. Next, the AR system may render a feature buffer exactly asdiscussed above at each of the three virtual keyframes.

Since the principal components are by definition orthogonal to eachother, rays drawn from each camera outwards may hit each other at apoint in 3D space. It should be appreciated that there may be multipleintersections of rays in some instances. Thus, there may now be Nfeatures in each virtual keyframe. Next, the AR system may use ageometric algorithm to find the points of intersection. This geometricalgorithm may be a constant time algorithm because there may be N³ rays.Next, masking and optimization may be performed in the same mannerdescribed above to find the map points in 3D space.

Referring to process flow diagram 3000 of FIG. 30, on a basic level, theAR system is configured to receive input (e.g., visual input 2202 fromthe user's wearable system, input from room cameras 2204, sensory input2206 in the form of various sensors in the system, gestures, totems, eyetracking etc.) from one or more AR systems. The AR systems mayconstitute one or more user wearable systems, and/or stationary roomsystems (room cameras, etc.). The wearable AR systems not only provideimages from the FOV cameras, they may also be equipped with varioussensors (e.g., accelerometers, temperature sensors, movement sensors,depth sensors, GPS, etc.), as discussed above, to determine thelocation, and various other attributes of the environment of the user.Of course, this information may further be supplemented with informationfrom stationary cameras discussed previously that may provide imagesand/or various cues from a different point of view. It should beappreciated that image data may be reduced to a set of points, asexplained above.

One or more object recognizers 2208 (object recognizers explained indepth above) crawl through the received data (e.g., the collection ofpoints) and recognize and/or map points with the help of the mappingdatabase 2210. The mapping database may comprise various pointscollected over time and their corresponding objects. It should beappreciated that the various devices, and the map database (similar tothe passable world) are all connected to each other through a network(e.g., LAN, WAN, etc.) to access the cloud.

Based on this information and collection of points in the map database,the object recognizers may recognize objects and supplement this withsemantic information (as explained above) to give life to the objects.For example, if the object recognizer recognizes a set of points to be adoor, the system may attach some semantic information (e.g., the doorhas a hinge and has a 90 degree movement about the hinge). Over time themap database grows as the system (which may reside locally or may beaccessible through a wireless network) accumulates more data from theworld.

Once the objects are recognized, the information may be transmitted toone or more user wearable systems 2220. For example, the system maytransmit information about a scene happening in California to one ormore users in New York. Based on data received from multiple FOV camerasand other inputs, the object recognizers and other software componentsmap the points collected through the various images, recognize objectsetc., such that the scene may be accurately “passed over” to a user in adifferent part of the world. As discussed above, the system may also usea topological map for localization purposes. More particularly, thefollowing discussion will go in depth about various elements of theoverall system that enables the interaction between one or more users ofthe AR system.

FIG. 31 is a process flow diagram 3100 that represents how a virtualscene may be represented to a user of the AR system. For example, theuser may be New York, but want to view a scene that is presently goingon in California, or may want to go on a walk with a friend who residesin California. First, in 2302, the AR system may receive input from theuser and other users regarding the environment of the user. As discussedpreviously, this may be achieved through various input devices, andknowledge already possessed in the map database.

The user's FOV cameras, sensors, GPS, eye tracking etc., conveysinformation to the system (step 2302). The system may then determinesparse points based on this information (step 2304). As discussed above,the sparse points may be used in determining pose data etc., that isimportant in displaying and understanding the orientation and positionof various objects in the user's surroundings. The object recognizersmay crawl through these collected points and recognize one or moreobjects using the map database (step 2306). This information may then beconveyed to the user's individual AR system (step 2308), and the desiredvirtual scene may be accordingly displayed to the user (step 2310). Forexample, the desired virtual scene (user in CA) may be displayed at theright orientation, position, etc., in relation to the various objectsand other surroundings of the user in New York. It should be appreciatedthat the above flow chart represents the system at a very basic level.FIG. 32 below represents a more detailed system architecture. It shouldbe appreciated that a number of user scenarios detailed below usesimilar processes as the one described above.

Referring now to FIG. 32, a more detailed system diagram 3200 isdescribed. As shown in FIG. 32, at the center of the system of a Map,which may be a database containing map data for the world. In oneembodiment it may partly reside on user-wearable components, and maypartly reside at cloud storage locations accessible by wired or wirelessnetwork. The Map (or the passable world model) is a significant andgrowing component which will become larger and larger as more and moreusers are on the system.

A Pose process may run on the wearable computing architecture andutilize data from the Map to determine position and orientation of thewearable computing hardware or user. Pose data may be computed from datacollected on the fly as the user is experiencing the system andoperating in the world. The data may comprise images, data from sensors(such as inertial measurement, or “IMU” devices, which generallycomprises accelerometer and gyro components), and surface informationpertinent to objects in the real or virtual environment.

What is known as a “sparse point representation” may be the output of asimultaneous localization and mapping (or “SLAM”; or “V-SLAM”, referringto a configuration wherein the input is images/visual only) process. Thesystem is configured to not only find out wherein the world the variouscomponents are, but what the world is made of. Pose is a building blockthat achieves many goals, including populating the Map and using thedata from the Map.

In one embodiment, sparse point position is not completely adequate onits own, and further information is needed to produce a multifocalvirtual or augmented reality experience as described above, which mayalso be termed “Cinematic Reality”. Dense Representations, generallyreferring to depth map information, may be utilized to fill this gap atleast in part. Such information may be computed from a process referredto as “Stereo”, wherein depth information is determined using atechnique such as triangulation or time-of-flight sensing.

Image information and active patterns (such as infrared patterns createdusing active projectors) may serve as input to the Stereo process. Asignificant amount of depth map information may be fused together, andsome of this may be summarized with surface representation. For example,mathematically definable surfaces are efficient (i.e., relative to alarge point cloud) and digestible inputs to things like game engines.

Thus the output of the Stereo process (depth map) may be combined in theFusion process. Pose may be an input to this Fusion process as well, andthe output of Fusion becomes an input to populating the Map process, asshown in the embodiment of FIG. 32. Sub-surfaces may connect with eachother, such as in topographical mapping, to form larger surfaces, andthe Map becomes a large hybrid of points and surfaces.

To resolve various aspects in a Cinematic Reality process, variousinputs may be utilized. For example, in the depicted embodiment, variousGame parameters may be inputs to determine that the user or operator ofthe system is playing a monster battling game with one or more monstersat various locations, monsters dying or running away under variousconditions (such as if the user shoots the monster), walls or otherobjects at various locations, and the like.

The Map may include information regarding where such objects arerelative to each other, to be another valuable input to CinematicReality. The input from the Map to the Cinematic Reality process may becalled the “World Map”. Pose relative to the world becomes an input aswell and plays a key role to almost any interactive system.

Controls or inputs from the user are another important input. Moredetails on various types of user inputs (e.g., visual input, gestures,totems, audio input, sensory input, etc.) will be described in furtherdetail below. In order to move around or play a game, for example, theuser may need to instruct the system regarding what he or she wants todo. Beyond just moving oneself in space, there are various forms of usercontrols that may be utilized. In one embodiment, a totem or object suchas a gun may be held by the user and tracked by the system.

The system preferably will be configured to know that the user isholding the item and understand what kind of interaction the user ishaving with the item (i.e., if the totem or object is a gun, the systemmay be configured to understand location and orientation, as well aswhether the user is clicking a trigger or other sensed button or elementwhich may be equipped with a sensor, such as an IMU, which may assist indetermining what is going on, even with such activity is not within thefield of view of any of the cameras.

Hand gesture tracking or recognition may also provide valuable inputinformation. The system may be configured to track and interpret handgestures for button presses, for gesturing left or right, stop, etc. Forexample, in one configuration, maybe the user wants to flip throughemails or your calendar in a non-gaming environment, or do a “fist bump”with another person or player.

The system may be configured to leverage a minimum amount of handgesture, which may or may not be dynamic. For example, the gestures maybe simple static gestures like open hand for stop, thumbs up for ok,thumbs down for not ok; or a hand flip right or left or up/down fordirectional commands. One embodiment may start with a fairly limitedvocabulary for gesture tracking and interpretation, and eventuallybecome more nuanced and complex.

Eye tracking is another important input (i.e., tracking where the useris looking to control the display technology to render at a specificdepth or range). In one embodiment, vergence of the eyes may bedetermined using triangulation, and then using a vergence/accommodationmodel developed for that particular person, accommodation may bedetermined.

With regard to the camera systems, the depicted configuration showsthree pairs of cameras: a relative wide field of view (“FOV”) or“passive SLAM” pair of cameras arranged to the sides of the user's face,a different pair of cameras oriented in front of the user to handle theStereo imaging process and also to capture hand gestures andtotem/object tracking in front of the user's face. Then there is a pairof Eye Cameras oriented into the eyes of the user so they may attempt totriangulate eye vectors and other information. As noted above, thesystem may also comprise one or more textured light projectors (such asinfrared, or “IR”, projectors) to inject texture into a scene.

Calibration of all of these devices (for example, the various cameras,IMU and other sensors, etc.) is important in coordinating the system andcomponents thereof. The system may also be configured to utilizewireless triangulation technologies (such as mobile wireless networktriangulation and/or global positioning satellite technology, both ofwhich become more relevant as the system is utilized outdoors). Otherdevices or inputs such as a pedometer worn by a user, a wheel encoderassociated with the location and/or orientation of the user, may need tobe calibrated to become valuable to the system.

The display system may also be considered to be an input element from acalibration perspective. In other words, the various elements of thesystem preferably are related to each other, and are calibratedintrinsically as well (i.e., how they map the real world matrix intomeasurements; going from real world measurements to matrix may be termed“intrinsics”; we want to know the intrinsics of the devices). For acamera module, the standard intrinsics parameters may include the focallength in pixels, the principal point (intersection of the optical axiswith the sensor), and distortion parameters (particularly geometry).

One may also want to consider photogrammetric parameters, ifnormalization of measurements or radiance in space is of interest. Withan IMU module that combines gyro and accelerometer devices, scalingfactors may be important calibration inputs. Camera-to-cameracalibration also may be key, and may be dealt with, at least in part, byhaving the three sets of cameras (Eye, Stereo, and World/wide FOV)rigidly coupled to each other.

In one embodiment, the display may have two eye sub-displays, which maybe calibrated at least partially in-factory, and partially in-situ dueto anatomic variations of the user (location of the eyes relative to theskull, location of the eyes relative to each other, etc.). Thus in oneembodiment, a process is conducted at runtime to calibrate the displaysystem for the particular user.

Generally all of the calibration will produce parameters orconfigurations which may be used as inputs to the other functionalblocks, as described above (for example: where are the cameras relativeto a helmet or other head-worn module; what is the global reference ofthe helmet; what are the intrinsic parameters of those cameras so thesystem can adjust the images on the fly—to know where every pixel in animage corresponds to in terms of ray direction in space; same with thestereo cameras; their disparity map may be mapped into a depth map, andinto an actual cloud of points in 3-D; so calibration is fundamentalthere as well; all of the cameras preferably will be known relative to asingle reference frame—a fundamental notion behind calibrating our headmounted system; same with the IMU—generally one will want to determinewhat the three axes of rotation are relative to the helmet, etc., tofacilitate at least some characterization/transformation relatedthereto.

The map described above is generated using various map points obtainedfrom multiple user devices. Various modes of collecting map points toadd on to the Map or the passable world model will be discussed below.

Dense/Sparse Mapping Tracking

As previously noted, there are many ways that one can obtain map pointsfor a given location, where some approaches may generate a large numberof (dense) points and other approaches may generate a much smallernumber of (sparse) points. However, conventional vision technologies arepremised upon the map data being one or the other.

This presents a problem when there is a need to have a single map thatcorresponds to both sparse and dense sets of data. For example, when inan indoor setting within a given space, there is often the need to storea very dense map of the point within the room, e.g., because the higherlevel and volume of detail for the points in the room is necessary tofulfill the requirements of many gaming or business applications. On theother hand, in an outdoor setting, there is far less need to store adense amount of data, and hence it may be far more efficient torepresent outdoor spaces using a sparse set of points.

With the wearable device of some embodiments of the invention, thesystem architecture is capable of accounting for the fact that the usermay move from a setting corresponding to a dense mapping (e.g., indoors)to a location corresponding to a sparse mapping (e.g., outdoors), andvice versa. The general idea is that regardless of the nature of theidentified point, certain information is obtained for that point, wherethese points are stored together into a common Map. A normalizationprocess is performed to make sure the stored information for the pointsis sufficient to allow the system to perform desired functionality forthe wearable device. This common Map therefore permits integration ofthe different types of data, and allows movement of the wearable devicewith seamless access and use of the Map data.

FIG. 33 shows a flowchart 3300 of one possible approach to populate theMap with both sparse map data and dense map data, where the path on theleft portion addresses sparse points and the path of the right portionaddresses dense points.

At 2401 a, the process identifies sparse feature points, which maypertain to any distinctive/repeatable visible to the machine. Examplesof such distinctive points include corners, circles, triangles, text,etc. Identification of these distinctive features allows one to identifyproperties for that point, and also to localize the identified point.Various type of information is obtained for the point, including thecoordinate of the point as well as other information pertaining to thecharacteristics of the texture of the region surrounding or adjacent tothe point.

Similarly, at 2401 b, identification is made of a large number of pointswithin a space. For example, a depth camera may be used to capture a setof 3D points within space that identifies the (x,y,z) coordinate of thatpoint. Some depth cameras may also capture the RGB values along with theD (depth) value for the points. This provides a set of world coordinatesfor the captured points.

The problem at this point is there are two sets of potentiallyincompatible points, where one set is sparse (resulting from 2401 a) andthe other set is dense (resulting from 2401 b). The present inventionperforms normalization on the captured data to address this potentialproblem. Normalization is performed to address any aspect of the datathat may be needed to facilitate vision functionality needed for thewearable device. For example, at 2403 a, scale normalization can beperformed to normalize the sparse data. Here, a point is identified, andoffsets from that point are also identified to determine differencesfrom the identified point to the offsets, where this process isperformed to check and determine the appropriate scaling that should beassociated with the point.

Similarly, at 2403 b, the dense data may also be normalized asappropriate to properly scale the identified dense points. Other typesof normalization may also be performed, e.g., coordinate normalizationto common origin point. A machine learning framework can be used toimplement the normalization process, so that the data obtained tonormalize a first point is used to normalize a second point, and so onuntil all necessary points have been normalized.

The normalized point data for both the sparse and dense points are thenrepresented in an appropriate data format. At 2405 a, a descriptor isgenerated and populated for each sparse point. Similarly, at 2405 b,descriptors are generated and populated for the dense points. Thedescriptors (e.g., using substantially the descriptor format for theA-KAZE algorithm) characterizes each of the points, whethercorresponding to sparse or dense data. For example, the descriptor mayinclude information about the scale, orientation, patch data, and/ortexture of the point. Thereafter, at 2407, the descriptors are thenstored into a common Map to unify the data, including both the sparseand dense data.

During operation of the wearable device, the data that is needed is usedby the system. For example, when the user is in a space corresponding todense data, a large number of points are likely available to perform anynecessary functionality using that data. On the other hand, when theuser has move to a location corresponding to sparse data, there may be alimited number of points that are used to perform the necessaryfunctionality. The user may be in an outdoor space where only threepoints are identified. The three points may be used, for example, forobject identification. The points may also be used to determine the poseof the user. For example, assume that the user has moved into a roomthat has already been mapped. The user's device will identify points inthe room (e.g., using a mono or stereo camera(s) on the wearabledevice). An attempt is made to check for the same points/patterns thatwere previously mapped, e.g., by identifying known points, the user'slocation can be identified as well as the user's orientation. Givenenough identified points in a 3D model of the room, this allows one todetermine the pose of the user. If there is a dense mapping, thenalgorithms appropriate for dense data can be used to make thedetermination. If the space corresponds to a sparse mapping, thenalgorithms appropriate for sparse data can be used to make thedetermination.

Projected Texture Sources

In some locations, there may be a scarcity of feature points from whichto obtain texture data for that space. For example, certain rooms mayhave wide swaths of blank walls for which there are no distinct featurepoints to identify to obtain the mapping data.

Some embodiments of the present invention provide a framework for veryefficiently and precisely describing the texture of a point, even in theabsence of distinct feature points. FIG. 34 illustrates an exampleapproach 3400 that can be taken to implement this aspect of embodimentsof the invention. One or more fiber-based projectors are employed toproject light that is visible to one or more cameras, such as camera 1and/or camera 2.

In one embodiment, the fiber-based projector comprises a scanned fiberdisplay scanner that projects a narrow beam of light back and forth atselected angles. The light may be projected through a lens or otheroptical element, which may be utilized to collect the angularly-scannedlight and convert it to one or more bundles of rays.

The projection data to be projected by the fiber-based projector maycomprise any suitable type of light. In some embodiments, the projectiondata comprises structured light having a series of dynamic knownpatterns, where successive light patterns are projected to identifyindividual pixels that can be individually addressed and textured. Theprojection data may also comprise patterned light having a pattern ofpoints to be identified and textured. In yet another embodiment, theprojection data comprises textured light, which does not necessarilyneed to comprise a recognizable pattern, but does include sufficienttexture to distinctly identify points within the light data.

In operation, the one or more camera(s) are placed having a recognizableoffset from the projector. The points are identified from the capturedimages from the one or more cameras, and triangulation is performed todetermine the requisite location and depth information for the point.With the textured light approach, the textured light permits one toidentify points even if there is already some texturing on the projectedsurface. This is implemented, for example, by having multiple camerasidentifying the same point from the projection (either from the texturedlight or from a real-world object), and then triangulating the correctlocation and depth information for that identified point. This isadvantageous over the structured light and patterned light approachesthat realistically need for the projected data to be identifiable.

Using the fiber-based projector for this functionality provides numerousadvantages. One advantage is that the fiber-based approach can be usedto draw light data exactly where it is desired for texturing purposes.This allows the system to place the visible point exactly where it needsto be projected and/or seen by the camera(s). In effect, this permits aperfectly controllable trigger for a triggerable texture source forgenerating the texture data. This allows the system o very quickly andeasily project and then find the desired point to be textured, and tothen triangulate its position and depth.

Another advantage provided by this approach is that some fiber-basedprojectors are also capable of capturing images. Therefore, in thisapproach, the cameras can be integrated into projector apparatus,providing savings in terms of cost, device real estate, and powerutilization. For example, when two fiber projectors/cameras are used,then this allows a first projector/camera to precisely project lightdata which is captured by the second projector/camera. Next, the reverseoccurs, where the second projector/camera precisely projects the lightdata to be captured by the first projector/camera. Triangulation canthen be performed for the captured data to generate texture informationfor the point.

As previously discussed, an AR system user may use a wearable structurehaving a display system positioned in front of the eyes of the user. Thedisplay is operatively coupled, such as by a wired lead or wirelessconnectivity, to a local processing and data module which may be mountedin a variety of configurations. The local processing and data module maycomprise a power-efficient processor or controller, as well as digitalmemory, such as flash memory, both of which may be utilized to assist inthe processing, caching, and storage of data a) captured from sensorswhich may be operatively coupled to the frame, such as image capturedevices (such as cameras), microphones, inertial measurement units,accelerometers, compasses, GPS units, radio devices, and/or gyros;and/or b) acquired and/or processed using a remote processing moduleand/or remote data repository, possibly for passage to the display aftersuch processing or retrieval.

The local processing and data module may be operatively coupled, such asvia a wired or wireless communication links, to the remote processingmodule and remote data repository such that these remote modules areoperatively coupled to each other and available as resources to thelocal processing and data module.

In some cloud-based embodiments, the remote processing module maycomprise one or more relatively powerful processors or controllersconfigured to analyze and process data and/or image information. FIG. 35depicts an example architecture 3500 that can be used in certaincloud-based computing embodiments. The cloud-based server(s) 3512 can beimplemented as one or more remote data repositories embodied as arelatively large-scale digital data storage facility, which may beavailable through the internet or other networking configuration in acloud resource configuration.

Various types of content can be stored in the cloud-based repository.For example, data collected on the fly as the user is experiencing thesystem and operating in the world. The data may comprise images, datafrom sensors (such as inertial measurement, or “IMU” devices, whichgenerally comprises accelerometer and gyro components), and surfaceinformation pertinent to objects in the real or virtual environment. Thesystem may generate various types of data and metadata from thecollected sensor data.

For example, geometry mapping data 3506 and semantic mapping data can begenerated and stored within the cloud-based repository. Map data is atype of data that can be cloud-based, which may be a database containingmap data for the world. In one embodiment, this data is entirely storedin the cloud. In another embodiment, this Map data partly resides onuser-wearable components, and may partly reside at cloud storagelocations accessible by wired or wireless network.

Cloud-based processing may be performed to process and/or analyze thedata. For example, the semantic map comprise information that providessematic content usable by the system, e.g., for objects and locations inthe world being tracked by the Map. One or more remote servers can beused to perform the processing (e.g., machine learning processing) toanalyze sensor data and to identify/generate the relevant semantic mapdata 3508. As another example, a Pose process may run to determineposition and orientation of the wearable computing hardware or user.

This Pose processing can also be performed on a remote server. In oneembodiment, the system processing is partially performed on cloud-basedservers and partially performed on processors in the wearable computingarchitecture. In an alternate embodiment, the entirety of the processingis performed on the remote servers. Any suitable partitioning of theworkload between the wearable device and the remote server (e.g.,cloud-based server) may be implemented, with consideration of thespecific work that is required, the relative available resources betweenthe wearable and the server, and the network bandwithavailability/requirements.

Cloud-based facilities may also be used to perform quality assuranceprocessing and error corrections for the stored data. Such tasks mayinclude, for example, error correction, labelling tasks, clean-upactivities, and generation of training data. Automaton can be used atthe remote server to perform these activities. Alternatively, remote“people resources” can also be employed, similar to the Mechanical Turkprogram provided by certain computing providers.

It should be appreciated that the mapping techniques (e.g., map pointcollection, pose determination, finding new map points, recognizingobjects based on map points, creating the map/passable world model,etc.) described above form the basis of how one or more users interactwith the AR system in their respective physical environments. Given thatthe AR system takes visual/audio/sensory data and converts it into mapdata to construct a virtual world or map of a virtual world that isstored in the cloud, the AR system is thus able to understand alocation, orientation, placement and configuration of physical objectsand can accordingly place virtual content in relation to the physicalworld.

This gives context and meaning to the virtual content that is generatedon the user device. For example, rather than haphazardly displayingvirtual content to the user (e.g., virtual content that is alwaysdisplayed on the top left side of the screen, etc.), the AR system maynow place virtual content at appropriate orientations/locations based onthe user's field of view. For example, virtual content may be displayedon top of various physical objects. Thus, rather than displaying amonster right in the middle of the screen, the monster may appear to bestanding on a physical object, for example. Mapping and knowing the realworld thus provides a huge advantage in strategically displaying virtualcontent in a meaningful manner, thereby greatly improving userexperience and interaction with the AR system.

Because the AR system is configured to continuously “know” the physicallocation and orientation of the user's surroundings, and given that theAR system is constantly collecting various types of data regarding theuser's environment (e.g., FOV images, eye tracking data, sensory data,audio data, etc.) conventional types of user inputs may not benecessary. For example, rather than the user physically pressing abutton or explicitly speaking a command, user input in the AR system maybe automatically recognized.

For example, the system may automatically recognize a gesture made bythe user's fingers. In another example, the AR system may recognize aninput based on eye tracking data. Or, in another example, the AR systemmay recognize a location, and automatically use that as user input todisplay virtual content. One important type of user input is gesturerecognition in order to perform an action or display virtual content, aswill be described below.

Gestures

In some implementations, the AR system may detect and be responsive toone or more finger/hand gestures. These gestures can take a variety offorms and may, for example, be based on inter-finger interaction,pointing, tapping, rubbing, etc. Other gestures may, for example,include 2D or 3D representations of characters (e.g., letters, digits,punctuation). To enter such, a user swipes their finger in the definedcharacter pattern. Other gestures may include thumb/wheel selection typegestures, which may, for example be used with a “popup” circular radialmenu which may be rendered in a field of view of a user, according toone illustrated embodiment.

Embodiments of the AR system can therefore recognize various commandsusing gestures, and in response perform certain functions mapped to thecommands. The mapping of gestures to commands may be universallydefined, across many users, facilitating development of variousapplications which employ at least some commonality in user interface.Alternatively or additionally, users or developers may define a mappingbetween at least some of the gestures and corresponding commands to beexecuted by the AR system in response to detection of the commands.

For example, a pointed index finger may indicate a command to focus, forexample to focus on a particular portion of a scene or virtual contentat which the index finger is pointed. A pinch gesture can be made withthe tip of the index finger touching a tip of the thumb to form a closedcircle, e.g., to indicate a grab and/or copy command. Another examplepinch gesture can be made with the tip of the ring finger touching a tipof the thumb to form a closed circle, e.g., to indicate a selectcommand.

Yet another example pinch gesture can be made with the tip of the pinkiefinger touching a tip of the thumb to form a closed circle, e.g., toindicate a back and/or cancel command. A gesture in which the ring andmiddle fingers are curled with the tip of the ring finger touching a tipof the thumb may indicate, for example, a click and/or menu command.Touching the tip of the index finger to a location on the head worncomponent or frame may indicate a return to home command.

Embodiments of the invention provide an advanced system and method forperforming gesture tracking and identification. In one embodiment, arejection cascade approach is performed, where multiple stages ofgesture analysis is performed upon image data to identify gestures. Asshown in the cascade 3600 of FIG. 36A, incoming images (e.g., an RGBimage at a depth D) is processed using a series of permissive analysisnodes. Each analysis node performed a distinct step of determiningwhether the image is identifiable as a gesture.

Each stage in this process performs a targeted computation so that thesequence of different in its totality can be used to efficiently performthe gesture processing. This means, for example, that the amount ofprocessing power at each stage of the process, along with thesequence/order of the nodes, can be used to optimize the ability toremove non-gestures while doing so with minimal computational expenses.For example, computationally less-expensive algorithms may be applied inthe earlier stages to remove large numbers of “easier” candidates,thereby leaving smaller numbers of “harder” data to be analyzed in laterstages using more computationally expensive algorithms.

The general approach to perform this type of processing in oneembodiment is shown in the flowchart 3601 of FIG. 36B. The first step isto generate candidates for the gesture processing (step 3602). Theseinclude, for example, images captured from sensor measurements of thewearable device, e.g., from camera(s) mounted on the wearable device.Next, analysis is performed on the candidates to generate analysis data(step 3604). For example, one type of analysis may be to check onwhether the contour of the shapes (e.g., fingers) in the image is sharpenough. Sorting is then performed on the analyzed candidates (step3606). Finally, any candidate that corresponds to a scoring/analysisvalue that is lower than a minimum threshold is removed fromconsideration (step 3608).

FIG. 36C depicts a more detailed approach 3650 for gesture analysisaccording to one embodiment of the invention. The first action is toperform depth segmentation upon the input data. For example, typicallythe camera providing the data inputs (e.g., the camera producingRGB+depth data) will be mounted on the user's head, where the camera FOV(field of view) will cover the range in which the human could reasonablyperform gestures. As shown in illustration 3660 of FIG. 36D, a linesearch can be performed through the data (e.g., from the bottom of theFOV).

If there are identifiable points along that line, then potentially agesture has been identified. Performing this analysis over a series oflines can be used to generate the depth data. In some embodiment, thistype of processing can be quite sparse—perhaps where 50 points areacquired relatively really quickly. Of course, different kinds of lineseries can be employed, e.g., in additional to or instead of flat linesacross the bottom, smaller diagonal lines are employed in the area wherethere might be a hand/arm.

Any suitable pattern may be employed, selecting ones that are mosteffective at detecting gestures. In some embodiments, aconfidence-enhanced depth map is obtained, where the data is floodfilled from cascade processing where a “flood feel” is performed tocheck for and filter whether the identified object is really a hand/arm.The confidence enhancement can be performed, for example, by getting aclear map of the hand and then checking for the amount of light isreflected off the hand in the images to the sensor, where the greateramount of light corresponds to a higher confidence level to enhance themap.

From the depth data, one can cascade to perform immediate/fastprocessing, e.g., where the image data is amenable to very fastrecognition of a gesture. This works best for very simple gesturesand/or hand/finger positions.

In many cases, deeper processing is to be performed to augment the depthmap. For example, one type of depth augmentation is to perform depthtransforms upon the data. Another type of augmentation is to check forgeodesic distances from specified point sets, such as boundaries,centroids, etc. For example, from a surface location, a determination ismade of the distance to various points on the map. This attempts tofind, for example, the farthest point to the tip of the fingers (byfinding the end of the fingers). The point sets may be from theboundaries (e.g., outline of hand) or centroid (e.g., statisticalcentral mass location).

Surface normalization may also be calculated. In addition, curvaturesmay also be estimated, which identifies how fast a contour turns—and toperform a filtering process to go over the points and removing concavepoints from fingers. In some embodiments, orientation normalization maybe performed on the data. To explain, consider that a given image of thehand may be captured with the hand in different positions. However, theanalysis may be expecting a canonical position of the image data of thehand. In this situation, as shown in illustration 3670 of FIG. 36E, themapped data may be re-oriented to change to a normalized/canonical handposition.

One advantageous approach in some embodiments of invention is to performbackground subtraction on the data. In many cases, a known backgroundexists in a scene, e.g., the pattern of a background wall. In thissituation, the map of the object to be analyzed can be enhanced byremoving the background image data. An example of this process is shownin illustration 3680 of FIG. 36F, where the left portion of the figureshows an image of a hand over some background image data. The right-handportion of the figure shows the results of removing the background fromthe image, leaving the augmented hand data with increased clarity andfocus.

Depth comparisons may also be performed upon points in the image toidentify the specific points that pertain to the hand (as opposed to thebackground non-hand data). For example, as shown in illustration 3690 ofFIG. 36G, it can be seen a first point A is located at a first depth anda second point B is located at a significantly different second depth.In this situation, the difference in the depths of these two pointsmakes it very evident that they likely belong to different objects.Therefore, if one knows that the depth of the hand is at the same depthvalue as point A, then one can conclude that point A is part of thehand. On the other hand, since the depth value for point B is not thesame as the depth of the hand, then one can readily conclude that pointB is not part of the hand.

At this point a series of analysis stages is performed upon the depthmap. Any number of analysis stages can be applied to the data. Thepresent embodiment shows three stages, but one of ordinary skill in theart would readily understand that any other number of stages (eithersmaller or larger) may be used as appropriate for the application towhich the invention is applied.

In the current embodiment, stage 1 analysis is performed using aclassifier mechanism upon the data. For example, aclassification/decision forest can be used to apply a series of yes/nodecisions in the analysis to identify the different parts of the handfor the different points in the mapping.

This identifies, for example, whether a particular point belongs to thepalm portion, back of hand, non-thumb finger, thumb, fingertip, and/orfinger joint. Any suitable classifier can be used for this analysisstage. For example, a deep learning module or a neural network mechanismcan be used instead of or in addition to the classification forest. Inaddition, a regression forest (e.g., using a Hough transformation) canbe used in addition to the classification forest.

The next stage of analysis (stage 2) can be used to further analysis themapping data. For example, analysis can be performed to identify jointlocations, articular, or to perform skeletonization on the data.Illustration 3695 of FIG. 36H provides an illustration ofskeletonization, where an original map of the hand data is used toidentify the locations of bones/joints within the hand, resulting in atype of “stick” figure model of the hand/hand skeleton. This type ofmodel provides with clarity a very distinct view of the location of thefingers and the specific orientation and/or configuration of the handcomponents. Labelling may also be applied at this stage to the differentparts of the hand.

At this point, it is possible that the data is now directly consumableby a downstream application without requiring any further analysis. Thusmay occur, for example, if the downstream application itself includeslogic to perform additional analysis/computations upon the model data.In addition, the system can also optionally cascade to performimmediate/fast processing, e.g., where the data is amenable to very fastrecognition of a gesture, such as the (1) first gesture; (2) open palmgesture; (3) finger gun gesture; (4) pinch; etc. For example, as shownin illustration 3698 of FIG. 36I, various points on the hand mapping(e.g., point on extended thumb and point on extended first finger) canbe used to immediately identify a pointing gesture. The outputs willthen proceed to a world engine, e.g., to take action upon a recognizedgesture.

In addition, deeper processing can be performed in the stage 3 analysis.This may involve, for example, using a decision forest/tree to classifythe gesture. This additional processing can be used to identify thegesture, determine a hand pose, identify context dependencies, and/orany other information as needed.

Prior/control information can be applied in any of the described stepsto optimize processing. This permits some biasing for the analysisactions taken in that stage of processing. For example, for gameprocessing, previous action taken in the game can be used to bias theanalysis based upon earlier hand positions/poses. In addition, aconfusion matrix can be used to more accurately perform the analysis.

Using the principles of gesture recognition discussed above, the ARsystem may use visual input gathered from the user's FOV cameras andrecognize various gestures that may be associated with a predeterminedcommand or action. Referring now to flowchart 3700 of FIG. 37, in step3102, the AR system may detect a gesture as discussed in detail above.

As described above, the movement of the fingers or a movement of thetotem may be compared to a database to detect a predetermined command,in step 3104. If a command is detected, the AR system determines thedesired action and/or desired virtual content based on the gesture, instep 3108. If the gesture or movement of the totem does not correspondto any known command, the AR system simply goes back to detecting othergestures or movements to step 3102.

In step 3108, the AR system determines the type of action necessary inorder to satisfy the command. For example, the user may want to switchan application, or may want to turn a page, may want to generate a userinterface, may want to connect to a friend located at another physicallocation, etc. Based on the desired action/virtual content, the ARsystem determines whether to retrieve information from the cloudservers, or whether the action can be performed using local resources onthe user device, in step 3110.

For example, if the user simply wants to turn a page of a virtual book,the required data may already have been downloaded or may resideentirely on the local device, in which case, the AR system simplyretrieves data associated with the next page and may display the nextpage to the user. Similarly, if the user wants to create a userinterface such that the user can draw a picture in the middle of space,the AR system may simply generate a virtual drawing surface in thedesired location without needing data from the cloud.

Data associated with many applications and capabilities may be stored onthe local device such that the user device does not need tounnecessarily connect to the cloud or access the passable world model.Thus, if the desired action can be performed locally, local data may beused to display virtual content corresponding to the detected gesture(step 3112).

Alternatively, in step 3114, if the system needs to retrieve data fromthe cloud or the passable world model, the system may send a request tothe cloud network, retrieve the appropriate data and send it back to thelocal device such that the action or virtual content may be appropriateddisplayed to the user. For example, if the user wants to connect to afriend at another physical location, the AR system may need to accessthe passable world model to retrieve the necessary data associated withthe physical form of the friend in order to render it accordingly at thelocal user device.

Thus, based on the user's interaction with the AR system, the AR systemmay create many types of user interfaces as desired by the user. Thefollowing represent some exemplary embodiments of user interfaces thatmay be created in a similar fashion to the exemplary process describedabove.

It should be appreciated that the above process is simplified forillustrative purposes, and other embodiments may include additionalsteps based on the desired user interface. The following discussion goesthrough various types of finger gestures, that may all be recognized andused such that the AR system automatically performs an action and/orpresents virtual content to the user that is either derived from thecloud or retrieved locally.

Finger Gestures

Finger gestures can take a variety of forms and may, for example, bebased on inter-finger interaction, pointing, tapping, rubbing, etc.

Other gestures may, for example, include 2D or 3D representations ofcharacters (e.g., letters, digits, punctuation). To enter such, a userswipes their finger in the defined character pattern. In oneimplementation of a user interface, the AR system renders three circles,each circle with specifically chosen characters (e.g., letters, digits,punctuation) arranged circumferentially around the periphery. The usercan swipe through the circles and letters to designate a characterselection or input. In another implementation, the AR system renders akeyboard (e.g., QWERTY keyboard) low in the user's field of view,proximate a position of the user's dominate hand in a bent-arm position.The user can than perform a swipe-like motion through desired keys, andthen indicate that the swipe gesture selection is complete by performinganother gesture (e.g., thumb-to-ring finger gesture) or otherproprioceptive interaction.

Other gestures may include thumb/wheel selection type gestures, whichmay, for example be used with a “popup” circular radial menu which maybe rendered in a field of view of a user, according to one illustratedembodiment.

Gestures 3800 of FIG. 38 shows a number of additional gestures. The ARsystem recognizes various commands, and in response performs certainfunctions mapped to the commands. The mapping of gestures to commandsmay be universally defined, across many users, facilitating developmentof various applications which employ at least some commonality in userinterface. Alternatively or additionally, users or developers may definea mapping between at least some of the gestures and correspondingcommands to be executed by the AR system in response to detection of thecommands.

In the top row left-most position, a pointed index finger may indicate acommand to focus, for example to focus on a particular portion of ascene or virtual content at which the index finger is pointed. In thetop row middle position, a first pinch gesture with the tip of the indexfinger touching a tip of the thumb to form a closed circle may indicatea grab and/or copy command. In the top row right-most position, a secondpinch gesture with the tip of the ring finger touching a tip of thethumb to form a closed circle may indicate a select command.

In the bottom row left-most position, a third pinch gesture with the tipof the pinkie finger touching a tip of the thumb to form a closed circlemay indicate a back and/or cancel command. In the bottom row middleposition, a gesture in which the ring and middle fingers are curled withthe tip of the ring finger touching a tip of the thumb may indicate aclick and/or menu command. In the bottom row right-most position,touching the tip of the index finger to a location on the head worncomponent or frame may indicate a return to home command. Such may causethe AR system to return to a home or default configuration, for exampledisplaying a home or default menu.

It should be appreciated that there may be many more types of user inputnot limited to the ones discussed above. For example, the system maymeasure neurological signals and use that as an input for the system.The system may have a sensor that tracks brain signals and map itagainst a table of commands. In other words, the user input is simplythe user's thoughts, that may be measured by the user's brain signals.This may also be referred to as subvocalization sensing. Such a systemmay also include apparatus for sensing EEG data to translate the user's“thoughts” into brain signals that may be decipherable by the system.

Totems

Similar to the above process where the AR system is configured torecognize various gestures and perform actions based on the gestures,the user may also use totems, or designated physical objects to controlthe AR system, or otherwise provide input to the system.

The AR system may detect or capture a user's interaction via tracking(e.g., visual tracking) of a totem. Numerous types of totems may beemployed in embodiments of the invention, including for example:

Existing Structures

Actively Marked Totems

Passively Marked Totems

Camera/Sensor Integration

Totem Controller Object

Any suitable existing physical structure can be used as a totem. Forexample, in gaming applications, a game object (e.g., tennis racket, guncontroller, etc.) can be recognized as a totem. One or more featurepoints can be recognized on the physical structure, providing a contextto identify the physical structure as a totem. Visual tracking can beperformed of the totem, employing one or more cameras to detect aposition, orientation, and/or movement (e.g., position, direction,distance, speed, acceleration) of the totem with respect to somereference frame (e.g., reference frame of a piece of media, the realworld, physical room, user's body, user's head).

Actively marked totems comprise some sort of active lighting or otherform of visual identification. Examples of such active marking include(a) flashing lights (e.g., LEDs); (b) lighted pattern groups; (c)reflective markers highlighted by lighting; (d) fiber-based lighting;(e) static light patterns; and/or (f) dynamic light patterns. Lightpatterns can be used to uniquely identify specific totems among multipletotems.

Passively marked totems comprise non-active lighting or identificationmeans. Examples of such passively marked totems include texturedpatterns and reflective markers.

The totem can also incorporate one or more cameras/sensors, so that noexternal equipment is need to track the totem. Instead, the totem willtrack itself and will provide its own location, orientation, and/oridentification to other devices. The on-board camera are used tovisually check for feature points, to perform visual tracking to detecta position, orientation, and/or movement (e.g., position, direction,distance, speed, acceleration) of the totem itself and with respect to areference frame. In addition, sensors mounted on the totem (such as aGPS sensor or accelerometers) can be used to detect the position andlocation of the totem.

A totem controller object is a device that can be mounted to anyphysical structure, and which incorporates functionality to facilitatetracking/identification of the totem. This allows any physical structureto become a totem merely by placing or affixing the totem controllerobject to that physical structure. The totem controller object may be apowered object that includes a battery to power electronics on theobject. The totem controller object may include communications, e.g.,wireless communications infrastructure such as an antenna and wirelessnetworking modem, to exchange messages with other devices. The totemcontroller object may also include any active marking (such as LEDs orfiber-based lighting), passive marking (such as reflectors or patterns),or cameras/sensors (such as cameras, GPS locator, or accelerometers).

As briefly described above, totems may be used to, for example, toprovide a virtual user interface. The AR system may, for example, rendera virtual user interface to appear on the totem.

The totem may take a large variety of forms. For example, the totem maybe an inanimate object. For instance, the totem may take the form of apiece or sheet of metal (e.g., aluminum). A processor component of anindividual AR system, for instance a belt pack, may serve as a totem.

The AR system may, for example, replicate a user interface of an actualphysical device (e.g., keyboard and/or trackpad of a computer, a mobilephone) on what is essentially a “dumb: totem. As an example, the ARsystem may render the user interface of an Android® phone onto a surfaceof an aluminum sheet. The AR system may detect interaction with therendered virtual user interface, for instance via a front facing camera,and implement functions based on the detected interactions.

For example, the AR system may implement one or more virtual actions,for instance render an updated display of Android® phone, render video,render display of a Webpage. Additionally or alternatively, the ARsystem may implement one or more actual or non-virtual actions, forinstance send email, send text, and/or place a phone call. This mayallow a user to select a desired user interface to interact with from aset of actual physical devices, for example various models of iPhones,iPads, Android based smartphones and/or tablets, or other smartphones,tablets, or even other types of appliances which have user interfacessuch as televisions, DVD/Blu-ray players, thermostats, etc.

Thus a totem may be any object on which virtual content can be rendered,including for example a body part (e.g., hand) to which virtual contentcan be locked in a user experience (UX) context. In someimplementations, the AR system can render virtual content so as toappear to be coming out from behind a totem, for instance appearing toemerge from behind a user's hand, and slowly wrapping at least partiallyaround the user's hand. The AR system detects user interaction with thevirtual content, for instance user finger manipulation with the virtualcontent which wrapped partially around the user's hand.

Alternatively, the AR system may render virtual content so as to appearto emerge from a palm of the user's hand, and detection user fingertipinteraction or manipulate of that virtual content. Thus, the virtualcontent may be locked to a reference from of a user's hand. The ARsystem may be responsive to various user interactions or gestures,including looking at some item of virtual content, moving hands,touching hands to themselves or to the environment, other gestures,opening and/or closing eyes, etc.

As described herein, the AR system may employ body center rendering,user center rendering, propreaceptic tactile interactions, pointing, eyevectors, totems, object recognizers, body sensor rendering, head posedetection, voice input, environment or ambient sound input, and theenvironment situation input.

Referring now to flowchart 3900 of FIG. 39, an exemplary process ofdetecting a user input through a totem is described. In step 2702, theAR system may detect a motion of a totem. It should be appreciated thatthe user may have already designated one or more physical objects as atotem during set-up, for example. The user may have multiple totems. Forexample, the user may have designated one totem for a social mediaapplication, another totem for playing games, etc. The movement of thetotem may be recognized through the user's FOV cameras, for example. Or,the movement may be detected through sensors (e.g., haptic glove, imagesensors, hand tracking devices, etc.) and captured.

Based on the detected and captured gesture or input through the totem,the AR system detects a position, orientation and/or movement of thetotem with respect to a reference frame, in step 2704. The referenceframe may be set of map points based on which the AR system translatesthe movement of the totem to an action or command. In step 2706, theuser's interaction with the totem is mapped. Based on the mapping of theuser interaction with respect to the reference frame 2704, the systemdetermines the user input.

For example, the user may move a totem or physical object back and forthto signify turning a virtual page and moving on to a next page. In orderto translate this movement with the totem the AR system may first needto recognize the totem as one that is routinely used for this purpose.For example, the user may use a playful wand on his desk to move it backand forth to signify turning a page.

The AR system, through sensors, or images captured of the wand, mayfirst detect the totem, and then use the movement of the wand withrespect to the reference frame to determine the input. For example, thereference frame, in this case may simply be a set of map pointsassociated with the stationary room. When the wand is moved back andforth, the map points of the wand change with respect to those of theroom, and a movement may thus be detected. This movement may then bemapped against a mapping database that is previously created todetermine the right command. For example, when the user first startsusing the user device, the system may calibrate certain movements anddefine them as certain commands.

For example, moving a wand back and forth for a width of at least 2inches may be a predetermined command to signify that the user wants toturn a virtual page. There may be a scoring system such that when themovement matches the predetermined gesture to a certain threshold value,the movement and the associated input is recognized, in one embodiment.When the detected movement matches a predetermined movement associatedwith a command stored in the map database, the AR system recognizes thecommand, and then performs the action desired by the user (e.g., displaythe next page to the user). The following discussion delves into variousphysical objects that may be used as totems, all of which use a similarprocess as the one described in FIG. 39.

FIG. 40 shows a totem 4012 according to one illustrated embodiment,which may be used as part of a virtual keyboard implementation. Thetotem may have generally rectangular profile and a soft durometersurface. The soft surface provides some tactile perception to a user asthe user interacts with the totem via touch.

As described above, the AR system may render the virtual keyboard imagein a user's field of view, such that the virtual keys, switches or otheruser input components appear to reside on the surface of the totem. TheAR system may, for example, render a 4D light field which is projecteddirectly to a user's retina. The 4D light field allows the user tovisually perceive the virtual keyboard with what appears to be realdepth.

The AR system may also detect or capture user interaction with thesurface of the totem. For example, the AR system may employ one or morefront facing cameras to detect a position and/or movement of a user'sfingers. In particularly, the AR system may identify from the capturedimages, any interactions of the user's fingers with various portions ofthe surface of the totem. The AR system maps the locations of thoseinteractions with the positions of virtual keys, and hence with variousinputs (e.g., characters, numbers, punctuation, controls, functions). Inresponse to the inputs, the AR system may cause the inputs to beprovided to a computer or some other device.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 41A shows a top surface of a totem 4014 according to oneillustrated embodiment, which may be used as part of a virtual mouseimplementation.

The top surface of the totem may have generally ovoid profile, with hardsurface portion, and one or more soft surface portions to replicate keysof a physical mouse. The soft surface portions do not actually need toimplement switches, and the totem may have no physical keys, physicalswitches or physical electronics. The soft surface portion(s) providessome tactile perception to a user as the user interacts with the totemvia touch.

The AR system may render the virtual mouse image in a user's field ofview, such that the virtual input structures (e.g., keys, buttons,scroll wheels, joystick, thumbstick) appear to reside on the top surfaceof the totem. The AR system may, for example, render a 4D light fieldwhich is projected directly to a user's retina to provide the visualperception of the virtual mouse with what appears to be real depth.Similar to the exemplary method outlined with reference to FIG. 39, theAR system may also detect or capture movement of the totem by the user,as well as, user interaction with the surface of the totem.

For example, the AR system may employ one or more front facing camerasto detect a position and/or movement of the mouse and/or interaction ofa user's fingers with the virtual input structures (e.g., keys). The ARsystem maps the position and/or movement of the mouse. The AR systemmaps user interactions with the positions of virtual input structures(e.g., keys), and hence with various inputs (e.g., controls, functions).In response to the position, movements and/or virtual input structureactivations, the AR system may cause corresponding inputs to be providedto a computer or some other device.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 41B shows a bottom surface of the totem 4016 of FIG. 41A, accordingto one illustrated embodiment, which may be used as part of a virtualtrackpad implementation.

The bottom surface of the totem may be flat with a generally oval orcircular profile. The bottom surface may be a hard surface. The totemmay have no physical input structures (e.g., keys, buttons, scrollwheels), no physical switches and no physical electronics.

The AR system may optionally render a virtual trackpad image in a user'sfield of view, such that the virtual demarcations appear to reside onthe bottom surface of the totem. Similar to the exemplary methodoutlined with reference to FIG. 39, the AR system detects or captures auser's interaction with the bottom surface of the totem. For example,the AR system may employ one or more front facing cameras to detect aposition and/or movement of a user's fingers on the bottom surface ofthe totem. For instance, the AR system may detect one or more staticpositions of one or more fingers, or a change in position of one or morefingers (e.g., swiping gesture with one or more fingers, pinchinggesture using two or more fingers).

The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap) of a user'sfingers with the bottom surface of the totem. The AR system maps theposition and/or movement (e.g., distance, direction, speed,acceleration) of the user's fingers along the bottom surface of thetotem. The AR system maps user interactions (e.g., number ofinteractions, types of interactions, duration of interactions) with thebottom surface of the totem, and hence with various inputs (e.g.,controls, functions). In response to the position, movements and/orinteractions, the AR system may cause corresponding inputs to beprovided to a computer or some other device.

FIG. 41C shows a top surface of a totem 4108 according to anotherillustrated embodiment, which may be used as part of a virtual mouseimplementation.

The totem of FIG. 41C is similar in many respects to that of the totemof FIG. 41A. Hence, similar or even identical structures are identifiedwith the same reference numbers. Only significant differences arediscussed below.

The top surface of the totem of FIG. 41C includes one or more indents ordepressions at one or more respective locations on the top surface wherethe AR system with render keys or other structures (e.g., scroll wheel)to appear. Operation of this virtual mouse is similar to the abovedescribed implementations of virtual mice.

FIG. 42A shows an orb totem 4020 with a flower petal-shaped (e.g., Lotusflower) virtual user interface according to another illustratedembodiment.

The totem may have a spherical shape with either a hard outer surface ora soft outer surface. The outer surface of the totem may have texture tofacilitate a sure grip by the user. The totem may have no physical keys,physical switches or physical electronics.

The AR system renders the flower petal-shaped virtual user interfaceimage in a user's field of view, so as to appear to be emanating fromthe totem. Each of the petals may correspond to a function, category offunctions, and/or category of content or media types, tools and/orapplications.

The AR system may optionally render one or more demarcations on theouter surface of the totem. Alternatively or additionally, the totem mayoptionally bear one or more physical demarcations (e.g., printed,inscribed) on the outer surface. The demarcation(s) may assist the userin visually orienting the totem with the flower petal-shaped virtualuser interface.

The AR system detects or captures a user's interaction with the totem.For example, the AR system may employ one or more front facing camerasto detect a position, orientation, and/or movement (e.g., rotationaldirection, magnitude of rotation, angular speed, angular acceleration)of the totem with respect to some reference frame (e.g., reference frameof the flower petal-shaped virtual user interface, real world, physicalroom, user's body, user's head) (similar to exemplary process flowdiagram of FIG. 39).

For instance, the AR system may detect one or more static orientationsor a change in orientation of the totem or a demarcation on the totem.The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap, fingertipgrip, enveloping grasp) of a user's fingers with outer surface of thetotem. The AR system maps the orientation and/or change in orientation(e.g., distance, direction, speed, acceleration) of the totem to userselections or inputs. The AR system optionally maps user interactions(e.g., number of interactions, types of interactions, duration ofinteractions) with the outer surface of the totem, and hence withvarious inputs (e.g., controls, functions). In response to theorientations, changes in position (e.g., movements) and/or interactions,the AR system may cause corresponding inputs to be provided to acomputer or some other device.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 42B shows an orb totem 4022 with a flower petal-shaped (e.g., Lotusflower) virtual user interface according to another illustratedembodiment.

The totem of FIG. 42B is similar in many respects to that of the totemof FIG. 42A. Hence, similar or even identical structures are identifiedwith the same reference numbers. Only significant differences arediscussed below.

The totem is disc shaped, having a top surface and bottom surface whichmay be flat or domed, as illustrated in FIG. 42B. That is a radius ofcurvature may be infinite or much larger than a radius of curvature of aperipheral edge of the totem.

The AR system renders the flower petal-shaped virtual user interfaceimage in a user's field of view, so as to appear to be emanating fromthe totem. As noted above, each of the petals may correspond to afunction, category of functions, and/or category of content or mediatypes, tools and/or applications.

Operation of this virtual mouse is similar to the above describedimplementations of virtual mice.

FIG. 42C shows an orb totem 4024 in a first configuration and a secondconfiguration, according to another illustrated embodiment.

In particular, the totem has a number of arms or elements which areselectively moveable or positionable with respect to each other. Forexample, a first arm or pair of arms may be rotated with respect to asecond arm or pair of arms. The first arm or pair of arms may be rotatedfrom a first configuration to a second configuration. Where the arms aregenerally arcuate, as illustrated, in the first configuration the armsform an orb or generally spherical structure. In the secondconfiguration, the second arm or pairs of arms align with the first armor pairs of arms to form an partial tube with a C-shaped profile.

The arms may have an inner diameter sized large enough to receive awrist or other limb of a user. The inner diameter may be sized smallenough to prevent the totem from sliding off the limb during use. Forexample, the inner diameter may be sized to comfortably receive a wristof a user, while not sliding past a hand of the user. This allows thetotem to take the form of a bracelet, for example when not in use, forconvenient carrying. A user may then configure the totem into an orbshape for use, in a fashion similar to the orb totems described above.The totem may have no physical keys, physical switches or physicalelectronics.

Notably, the virtual user interface is omitted from FIG. 42C. The ARsystem may render a virtual user interface in any of a large variety offorms, for example the flower petal-shaped virtual user interfacepreviously illustrated and discussed.

FIG. 43A shows a handheld controller shaped totem 4026, according toanother illustrated embodiment. The totem has a gripping section sizedand configured to comfortably fit in a user's hand. The totem mayinclude a number of user input elements, for example a key or button anda scroll wheel. The user input elements may be physical elements,although not connected to any sensor or switches in the totem, whichitself may have no physical switches or physical electronics.

Alternatively, the user input elements may be virtual elements renderedby the AR system. Where the user input elements are virtual elements,the totem may have depressions, cavities, protrusions, textures or otherstructures to tactile replicate a feel of the user input element.

The AR system detects or captures a user's interaction with the userinput elements of the totem. For example, the AR system may employ oneor more front facing cameras to detect a position and/or movement of auser's fingers with respect to the user input elements of the totem(similar to exemplary process flow diagram of FIG. 39). For instance,the AR system may detect one or more static positions of one or morefingers, or a change in position of one or more fingers (e.g., swipingor rocking gesture with one or more fingers, rotating or scrollinggesture, or both).

The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap) of a user'sfingers with the user input elements of the totem. The AR system mapsthe position and/or movement (e.g., distance, direction, speed,acceleration) of the user's fingers with the user input elements of thetotem. The AR system maps user interactions (e.g., number ofinteractions, types of interactions, duration of interactions) of theuser's fingers with the user input elements of the totem, and hence withvarious inputs (e.g., controls, functions). In response to the position,movements and/or interactions, the AR system may cause correspondinginputs to be provided to a computer or some other device.

FIG. 43B shows a block shaped totem 4028, according to anotherillustrated embodiment. The totem may have the shape of a cube with sixfaces, or some other three-dimensional geometric structure. The totemmay have a hard outer surface or a soft outer surface. The outer surfaceof the totem may have texture to facilitate a sure grip by the user. Thetotem may have no physical keys, physical switches or physicalelectronics.

The AR system renders a virtual user interface image in a user's fieldof view, so as to appear to be on the face(s) of the outer surface ofthe totem. Each of the faces, and corresponding virtual input prompt,may correspond to a function, category of functions, and/or category ofcontent or media types, tools and/or applications.

The AR system detects or captures a user's interaction with the totem.For example, the AR system may employ one or more front facing camerasto detect a position, orientation, and/or movement (e.g., rotationaldirection, magnitude of rotation, angular speed, angular acceleration)of the totem with respect to some reference frame (e.g., reference frameof the real world, physical room, user's body, user's head) (similar toexemplary process flow diagram of FIG. 39). For instance, the AR systemmay detect one or more static orientations or a change in orientation ofthe totem.

The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap, fingertipgrip, enveloping grasp) of a user's fingers with outer surface of thetotem. The AR system maps the orientation and/or change in orientation(e.g., distance, direction, speed, acceleration) of the totem to userselections or inputs. The AR system optionally maps user interactions(e.g., number of interactions, types of interactions, duration ofinteractions) with the outer surface of the totem, and hence withvarious inputs (e.g., controls, functions). In response to theorientations, changes in position (e.g., movements) and/or interactions,the AR system may cause corresponding inputs to be provided to acomputer or some other device.

In response to the orientations, changes in position (e.g., movements)and/or interactions, the AR system may change one or more aspects of therendering the virtual user interface cause corresponding inputs to beprovided to a computer or some other device. For example, as a userrotates the totem, different faces may come into the user's field ofview, while other faces rotate out of the user's field of view. The ARsystem may respond by rendering virtual interface elements to appear onthe now visible faces, which were previously hidden from the view of theuser. Likewise, the AR system may respond by stopping the rendering ofvirtual interface elements which would otherwise appear on the faces nowhidden from the view of the user.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 43C shows a handheld controller shaped totem 4030, according toanother illustrated embodiment. The totem has a gripping section sizedand configured to comfortably fit in a user's hand, for example acylindrically tubular portion. The totem may include a number of userinput elements, for example a number of pressure sensitive switches anda joy or thumbstick. The user input elements may be physical elements,although not connected to any sensor or switches in the totem, whichitself may have no physical switches or physical electronics.

Alternatively, the user input elements may be virtual elements renderedby the AR system. Where the user input elements are virtual elements,the totem may have depressions, cavities, protrusions, textures or otherstructures to tactile replicate a feel of the user input element.

The AR system detects or captures a user's interaction with the userinput elements of the totem. For example, the AR system may employ oneor more front facing cameras to detect a position and/or movement of auser's fingers with respect to the user input elements of the totem(similar to exemplary process flow diagram of FIG. 39). For instance,the AR system may detect one or more static positions of one or morefingers, or a change in position of one or more fingers (e.g., swipingor rocking gesture with one or more fingers, rotating or scrollinggesture, or both).

The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap) of a user'sfingers with the user input elements of the totem. The AR system mapsthe position and/or movement (e.g., distance, direction, speed,acceleration) of the user's fingers with the user input elements of thetotem. The AR system maps user interactions (e.g., number ofinteractions, types of interactions, duration of interactions) of theuser's fingers with the user input elements of the totem, and hence withvarious inputs (e.g., controls, functions). In response to the position,movements and/or interactions, the AR system may cause correspondinginputs to be provided to a computer or some other device.

FIG. 43D shows a handheld controller shaped totem 4032, according toanother illustrated embodiment. The totem has a gripping section sizedand configured to comfortably fit in a user's hand. The totem mayinclude a number of user input elements, for example a key or button anda joy or thumbstick. The user input elements may be physical elements,although not connected to any sensor or switches in the totem, whichitself may have no physical switches or physical electronics.Alternatively, the user input elements may be virtual elements renderedby the AR system. Where the user input elements are virtual elements,the totem may have depressions, cavities, protrusions, textures or otherstructures to tactile replicate a feel of the user input element.

The AR system detects or captures a user's interaction with the userinput elements of the totem. For example, the AR system may employ oneor more front facing cameras to detect a position and/or movement of auser's fingers with respect to the user input elements of the totem(similar to exemplary process flow diagram of FIG. 39). For instance,the AR system may detect one or more static positions of one or morefingers, or a change in position of one or more fingers (e.g., swipingor rocking gesture with one or more fingers, rotating or scrollinggesture, or both). The AR system may also employ the front facingcamera(s) to detect interactions (e.g., tap, double tap, short tap, longtap) of a user's fingers with the user input elements of the totem.

The AR system maps the position and/or movement (e.g., distance,direction, speed, acceleration) of the user's fingers with the userinput elements of the totem. The AR system maps user interactions (e.g.,number of interactions, types of interactions, duration of interactions)of the user's fingers with the user input elements of the totem, andhence with various inputs (e.g., controls, functions). In response tothe position, movements and/or interactions, the AR system may causecorresponding inputs to be provided to a computer or some other device.

FIG. 44A shows a ring totem 4034, according one illustrated embodiment.In particular, the ring totem has a tubular portion and an interactionportion physically coupled to the tubular portion. The tubular andinteraction portions may be integral, and may be formed as or from asingle unitary structure. The tubular portion has an inner diametersized large enough to receive a finger of a user therethrough. The innerdiameter may be sized small enough to prevent the totem from sliding offthe finger during normal use. This allows the ring totem to becomfortably worn even when not in active use, ensuring availability whenneeded. The ring totem may have no physical keys, physical switches orphysical electronics.

The AR system may render a virtual user interface in any of a largevariety of forms. For example, the AR system may render a virtual userinterface in the user's field of view as to appear as if the virtualuser interface element(s) reside on the interaction surface.Alternatively, the AR system may render a virtual user interface as theflower petal-shaped virtual user interface previously illustrated anddiscussed, emanating from the interaction surface.

The AR system detects or captures a user's interaction with the totem.For example, the AR system may employ one or more front facing camerasto detect a position, orientation, and/or movement (e.g., position,direction, distance, speed, acceleration) of the user's finger(s) withrespect to interaction surface in some reference frame (e.g., referenceframe of the interaction surface, real world, physical room, user'sbody, user's head) (similar to exemplary process flow diagram of FIG.39). For instance, the AR system may detect one or more locations oftouches or a change in position of a finger on the interaction surface.

The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap, fingertipgrip, enveloping grasp) of a user's fingers with the interaction surfaceof the totem. The AR system maps the position, orientation, and/ormovement of the finger with respect to the interaction surface to a setof user selections or inputs. The AR system optionally maps other userinteractions (e.g., number of interactions, types of interactions,duration of interactions) with the interaction surface of the totem, andhence with various inputs (e.g., controls, functions). In response tothe position, orientation, movement, and/or other interactions, the ARsystem may cause corresponding inputs to be provided to a computer orsome other device.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 44B shows a bracelet totem 4036, according one illustratedembodiment. In particular, the bracelet totem has a tubular portion anda touch surface physically coupled to the tubular portion. The tubularportion and touch surface may be integral, and may be formed as or froma single unitary structure. The tubular portion has an inner diametersized large enough to receive a wrist or other limb of a user.

The inner diameter may be sized small enough to prevent the totem fromsliding off the limb during use. For example, the inner diameter may besized to comfortably receive a wrist of a user, while not sliding past ahand of the user. This allows the bracelet totem to be worn whether inactive use or not, ensuring availability when desired. The bracelettotem may have no physical keys, physical switches or physicalelectronics.

The AR system may render a virtual user interface in any of a largevariety of forms. For example, the AR system may render a virtual userinterface in the user's field of view as to appear as if the virtualuser interface element(s) reside on the touch surface. Alternatively,the AR system may render a virtual user interface as the flowerpetal-shaped virtual user interface previously illustrated anddiscussed, emanating from the touch surface.

The AR system detects or captures a user's interaction with the totem(similar to exemplary process flow diagram of FIG. 39). For example, theAR system may employ one or more front facing cameras to detect aposition, orientation, and/or movement (e.g., position, direction,distance, speed, acceleration) of the user's finger(s) with respect totouch surface in some reference frame (e.g., reference frame of thetouch surface, real world, physical room, user's body, user's head). Forinstance, the AR system may detect one or more locations of touches or achange in position of a finger on the touch surface.

The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap, fingertipgrip, enveloping grasp) of a user's fingers with the touch surface ofthe totem. The AR system maps the position, orientation, and/or movementof the finger with respect to the touch surface to a set of userselections or inputs. The AR system optionally maps other userinteractions (e.g., number of interactions, types of interactions,duration of interactions) with the touch surface of the totem, and hencewith various inputs (e.g., controls, functions). In response to theposition, orientation, movement, and/or other interactions, the ARsystem may cause corresponding inputs to be provided to a computer orsome other device.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 44C shows a ring totem 4038, according another illustratedembodiment. In particular, the ring totem has a tubular portion and aninteraction portion physically rotatably coupled to the tubular portionto rotate with respect thereto. The tubular portion has an innerdiameter sized large enough to receive a finger of a user there through.The inner diameter may be sized small enough to prevent the totem fromsliding off the finger during normal use.

This allows the ring totem to be comfortably worn even when not inactive use, ensuring availability when needed. The interaction portionmay itself be a closed tubular member, having a respective innerdiameter received about an outer diameter of the tubular portion. Forexample, the interaction portion may be journaled or slideable mountedto the tubular portion. The interaction portion is accessible from anexterior surface of the ring totem. The interaction portion may, forexample, be rotatable in a first rotational direction about alongitudinal axis of the tubular portion. The interaction portion mayadditionally be rotatable in a second rotational, opposite the firstrotational direction about the longitudinal axis of the tubular portion.The ring totem may have no physical switches or physical electronics.

The AR system may render a virtual user interface in any of a largevariety of forms. For example, the AR system may render a virtual userinterface in the user's field of view as to appear as if the virtualuser interface element(s) reside on the interaction portion.Alternatively, the AR system may render a virtual user interface as theflower petal-shaped virtual user interface previously illustrated anddiscussed, emanating from the interaction portion.

The AR system detects or captures a user's interaction with the totem(similar to exemplary process flow diagram of FIG. 39). For example, theAR system may employ one or more front facing cameras to detect aposition, orientation, and/or movement (e.g., position, direction,distance, speed, acceleration) of the interaction portion with respectto the tubular portion (e.g., finger receiving portion) in somereference frame (e.g., reference frame of the tubular portion, realworld, physical room, user's body, user's head).

For instance, the AR system may detect one or more locations ororientations or changes in position or orientation of the interactionportion with respect to the tubular portion. The AR system may alsoemploy the front facing camera(s) to detect interactions (e.g., tap,double tap, short tap, long tap, fingertip grip, enveloping grasp) of auser's fingers with the interaction portion of the totem.

The AR system maps the position, orientation, and/or movement of theinteraction portion with respect the tubular portion to a set of userselections or inputs. The AR system optionally maps other userinteractions (e.g., number of interactions, types of interactions,duration of interactions) with the interaction portion of the totem, andhence with various inputs (e.g., controls, functions). In response tothe position, orientation, movement, and/or other interactions, the ARsystem may cause corresponding inputs to be provided to a computer orsome other device.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 45A shows a glove-shaped haptic totem 4040, according oneillustrated embodiment. In particular, the glove-shaped haptic totem isshaped like a glove or partial glove, having an opening for receiving awrist and one or more tubular glove fingers (three shown) sized toreceive a user's fingers. The glove-shaped haptic totem may be made ofone or more of a variety of materials. The materials may be elastomericor may otherwise conform the shape or contours of a user's hand,providing a snug but comfortable fit.

The bracelet totem may have no physical keys, physical switches orphysical electronics. The AR system may render a virtual user interfacein any of a large variety of forms. For example, the AR system mayrender a virtual user interface in the user's field of view as to appearas if the virtual user interface element(s) is inter-actable via theglove-shaped haptic totem. For example, the AR system may render avirtual user interface as one of the previously illustrated and/ordescribed totems or virtual user interfaces.

The AR system detects or captures a user's interaction via visualtracking of the user's hand and fingers on which the glove-shaped haptictotem is worn (similar to exemplary process flow diagram of FIG. 39).For example, the AR system may employ one or more front facing camerasto detect a position, orientation, and/or movement (e.g., position,direction, distance, speed, acceleration) of the user's hand and/orfinger(s) with respect to some reference frame (e.g., reference frame ofthe touch surface, real world, physical room, user's body, user's head).

For instance, the AR system may detect one or more locations of touchesor a change in position of a hand and/or fingers. The AR system may alsoemploy the front facing camera(s) to detect interactions (e.g., tap,double tap, short tap, long tap, fingertip grip, enveloping grasp) of auser's hands and/or fingers. Notably, the AR system may track theglove-shaped haptic totem instead of the user's hands and fingers. TheAR system maps the position, orientation, and/or movement of the handand/or fingers to a set of user selections or inputs.

The AR system optionally maps other user interactions (e.g., number ofinteractions, types of interactions, duration of interactions), andhence with various inputs (e.g., controls, functions). In response tothe position, orientation, movement, and/or other interactions, the ARsystem may cause corresponding inputs to be provided to a computer orsome other device.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a new set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

The glove-shaped haptic totem includes a plurality of actuators, whichare responsive to signals to provide haptic sensations such as pressureand texture. The actuators may take any of a large variety of forms, forexample piezoelectric elements, and/or micro electrical mechanicalstructures (MEMS).

The AR system provides haptic feedback to the user via the glove-shapedhaptic totem. In particular, the AR system provides signals to theglove-shaped haptic totem to replicate a sensory sensation ofinteracting with a physical object which a virtual object may represent.Such may include providing a sense of pressure and/or texture associatedwith a physical object.

Thus, the AR system may cause a user to feel a presence of a virtualobject, for example including various structural features of thephysical object such as edges, corners, roundness, etc. The AR systemmay also cause a user to feel textures such as smooth, rough, dimpled,etc.

FIG. 45B shows a stylus or brush shaped totem 4042, according oneillustrated embodiment. The stylus or brush shaped totem includes anelongated handle, similar to that of any number of conventional stylusor brush. In contrast to conventional stylus or brush, the stylus orbrush has a virtual tip or bristles.

In particular, the AR system may render a desired style of virtual tipor bristle to appear at an end of the physical stylus or brush. The tipor bristle may take any conventional style including narrow or widepoints, flat bristle brushed, tapered, slanted or cut bristle brushed,natural fiber bristle brushes (e.g., horse hair), artificial fiberbristle brushes, etc. Such advantageously allows the virtual tip orbristles to be replaceable.

The AR system detects or captures a user's interaction via visualtracking of the user's hand and/or fingers on the stylus or brush and/orvia visual tracking of the end of the stylus or brush (similar toexemplary process flow diagram of FIG. 39). For example, the AR systemmay employ one or more front facing cameras to detect a position,orientation, and/or movement (e.g., position, direction, distance,speed, acceleration) of the user's hand and/or finger(s) and/or end ofthe stylus or brush with respect to some reference frame (e.g.,reference frame of a piece of media, the real world, physical room,user's body, user's head).

For instance, the AR system may detect one or more locations of touchesor a change in position of a hand and/or fingers. Also for instance, theAR system may detect one or more locations of the end of the stylus orbrush and/or an orientation of the end of the stylus or brush withrespect to, for example, a piece of media or totem representing a pieceof media.

The AR system may additionally or alternatively detect one or morechange in locations of the end of the stylus or brush and/or change inorientation of the end of the stylus or brush with respect to, forexample, the piece of media or totem representing the piece of media.The AR system may also employ the front facing camera(s) to detectinteractions (e.g., tap, double tap, short tap, long tap, fingertipgrip, enveloping grasp) of a user's hands and/or fingers or of thestylus or brush.

The AR system maps the position, orientation, and/or movement of thehand and/or fingers and/or end of the stylus or brush to a set of userselections or inputs. The AR system optionally maps other userinteractions (e.g., number of interactions, types of interactions,duration of interactions), and hence with various inputs (e.g.,controls, functions). In response to the position, orientation,movement, and/or other interactions, the AR system may causecorresponding inputs to be provided to a computer or some other device.

Additionally or alternatively, the AR system may render a virtual imageof markings made by the user using the stylus or brush, taking intoaccount the visual effects that would be achieved by the selected tip orbristles.

The stylus or brush may have one or more haptic elements (e.g.,piezoelectric elements, MEMS elements), which the AR system control toprovide a sensation (e.g., smooth, rough, low friction, high friction)that replicate a feel of a selected point or bristles, as the selectedpoint or bristles pass over media. The sensation may also reflect orreplicate how the end or bristles would interact with different types ofphysical aspects of the media, which may be selected by the user. Thus,paper and canvass may produce two different haptic responses.

FIG. 45C shows a pen shaped totem 4044, according one illustratedembodiment.

The pen shaped totem includes an elongated shaft, similar to that of anynumber of conventional pen, pencil, stylus or brush. The pen shapedtotem has a user actuatable joy or thumbstick located at one end of theshaft. The joy or thumbstick is moveable with respect to the elongatedshaft in response to user actuation.

The joy or thumbstick may, for example, be pivotally movable in fourdirections (e.g., forward, back, left, right). Alternatively, the joy orthumbstick may, for example, be movable in all directions fourdirections, or may be pivotally moveable in any angular direction in acircle, for example to navigate. Notably, the joy or thumbstick is notcoupled to any switch or electronics.

Instead of coupling the joy or thumbstick to a switch or electronics,the AR system detects or captures a position, orientation, or movementof the joy or thumbstick. For example, the AR system may employ one ormore front facing cameras to detect a position, orientation, and/ormovement (e.g., position, direction, distance, speed, acceleration) ofthe joy or thumbstick with respect to some reference frame (e.g.,reference frame of the elongated shaft.

Additionally, the AR system may employ one or more front facing camerasto detect a position, orientation, and/or movement (e.g., position,direction, distance, speed, acceleration) of the user's hand and/orfinger(s) and/or end of the pen shaped totem with respect to somereference frame (e.g., reference frame of the elongated shaft, of apiece of media, the real world, physical room, user's body, user's head)(similar to exemplary process flow diagram of FIG. 39). For instance,the AR system may detect one or more locations of touches or a change inposition of a hand and/or fingers.

Also for instance, the AR system may detect one or more locations of theend of the pen shaped totem and/or an orientation of the end of the penshaped totem with respect to, for example, a piece of media or totemrepresenting a piece of media. The AR system may additionally oralternatively detect one or more change in locations of the end of thepen shaped totem and/or change in orientation of the end of the penshaped totem with respect to, for example, the piece of media or totemrepresenting the piece of media. The AR system may also employ the frontfacing camera(s) to detect interactions (e.g., tap, double tap, shorttap, long tap, fingertip grip, enveloping grasp) of a user's handsand/or fingers with the joy or thumbstick or the elongated shaft of thepen shaped totem.

The AR system maps the position, orientation, and/or movement of thehand and/or fingers and/or end of the joy or thumbstick to a set of userselections or inputs. The AR system optionally maps other userinteractions (e.g., number of interactions, types of interactions,duration of interactions), and hence with various inputs (e.g.,controls, functions). In response to the position, orientation,movement, and/or other interactions, the AR system may causecorresponding inputs to be provided to a computer or some other device.

Additionally or alternatively, the AR system may render a virtual imageof markings made by the user using the pen shaped totem, taking intoaccount the visual effects that would be achieved by the selected tip orbristles.

The pen shaped totem may have one or more haptic elements (e.g.,piezoelectric elements, MEMS elements), which the AR system control toprovide a sensation (e.g., smooth, rough, low friction, high friction)that replicate a feel of passing over media.

FIG. 46A shows a charm chain totem 4046, according one illustratedembodiment.

The charm chain totem includes a chain and a number of charms. The chainmay include a plurality of interconnected links which providesflexibility to the chain. The chain may also include a closure or claspwhich allows opposite ends of the chain to be securely coupled together.The chain and/or clasp may take a large variety of forms, for examplesingle strand, multi-strand, links or braided. The chain and/or claspmay be formed of any variety of metals, or other non-metallic materials.

A length of the chain should accommodate a portion of a user's limb whenthe two ends are clasped together. The length of the chain should alsobe sized to ensure that the chain is retained, even loosely, on theportion of the limb when the two ends are clasped together. The chainmay be worn as a bracket on a wrist of an arm or on an ankle of a leg.The chain may be worn as a necklace about a neck.

The charms may take any of a large variety of forms. The charms may havea variety of shapes, although will typically take the form of plates ordiscs. While illustrated with generally rectangular profiles, the charmsmay have any variety of profiles, and different charms on a single chainmay have respective profiles which differ from one another. The charmsmay be formed of any of a large variety of metals, or non-metallicmaterials.

Each charm may bear an indicia, which is logically associable in atleast one computer- or processor-readable non-transitory storage mediumwith a function, category of functions, category of content or mediatypes, and/or tools or applications which is accessible via the ARsystem.

Adding on the exemplary method of using totems described in FIG. 39, theAR system may detect or captures a user's interaction with the charms ofFIG. 46A. For example, the AR system may employ one or more front facingcameras to detect touching or manipulation of the charms by the user'sfingers or hands. For instance, the AR system may detect a selection ofa particular charm by the user touching the respective charm with theirfinger or grasping the respective charm with two or more fingers.

Further, the augmented reality may detect a position, orientation,and/or movement (e.g., rotational direction, magnitude of rotation,angular speed, angular acceleration) of a charm with respect to somereference frame (e.g., reference frame of the portion of the body, realworld, physical room, user's body, user's head). The AR system may alsoemploy the front facing camera(s) to detect other interactions (e.g.,tap, double tap, short tap, long tap, fingertip grip, enveloping grasp)of a user's fingers with a charm. The AR system maps selection of thecharm to user selections or inputs, for instance selection of a socialmedia application.

The AR system optionally maps other user interactions (e.g., number ofinteractions, types of interactions, duration of interactions) with thecharm, and hence with various inputs (e.g., controls, functions) withthe corresponding application. In response to the touching, manipulationor other interactions with the charms, the AR system may causecorresponding applications to be activated and/or provide correspondinginputs to the applications.

Additionally or alternatively, the AR system may render the virtual userinterface differently in response to select user interactions. Forinstance, some user interactions may correspond to selection of aparticular submenu, application or function. The AR system may respondto such selection by rendering a set of virtual interface elements,based at least in part on the selection. For instance, the AR systemrender a submenu or a menu or other virtual interface element associatedwith the selected application or functions. Thus, the rendering by ARsystem may be context sensitive.

FIG. 46B shows a keychain totem 4048, according one illustratedembodiment. The keychain totem includes a chain and a number of keys.The chain may include a plurality of interconnected links which providesflexibility to the chain. The chain may also include a closure or claspwhich allows opposite ends of the chain to be securely coupled together.The chain and/or clasp may take a large variety of forms, for examplesingle strand, multi-strand, links or braided. The chain and/or claspmay be formed of any variety of metals, or other non-metallic materials.

The keys may take any of a large variety of forms. The keys may have avariety of shapes, although will typically take the form of conventionalkeys, either with or without ridges and valleys (e.g., teeth). In someimplementations, the keys may open corresponding mechanical locks, whilein other implementations the keys only function as totems and do notopen mechanical locks. The keys may have any variety of profiles, anddifferent keys on a single chain may have respective profiles whichdiffer from one another. The keys may be formed of any of a largevariety of metals, or non-metallic materials. Various keys may bedifferent colors from one another.

Each key may bear an indicia, which is logically associable in at leastone computer- or processor-readable non-transitory storage medium with afunction, category of functions, category of content or media types,and/or tools or applications which is accessible via the AR system.

The AR system detects or captures a user's interaction with the keys(similar to exemplary process flow diagram of FIG. 39). For example, theAR system may employ one or more front facing cameras to detect touchingor manipulation of the keys by the user's fingers or hands. Forinstance, the AR system may detect a selection of a particular key bythe user touching the respective key with their finger or grasping therespective key with two or more fingers.

Further, the augmented reality may detect a position, orientation,and/or movement (e.g., rotational direction, magnitude of rotation,angular speed, angular acceleration) of a key with respect to somereference frame (e.g., reference frame of the portion of the body, realworld, physical room, user's body, user's head). The AR system may alsoemploy the front facing camera(s) to detect other interactions (e.g.,tap, double tap, short tap, long tap, fingertip grip, enveloping grasp)of a user's fingers with a key.

The AR system maps selection of the key to user selections or inputs,for instance selection of a social media application. The AR systemoptionally maps other user interactions (e.g., number of interactions,types of interactions, duration of interactions) with the key, and hencewith various inputs (e.g., controls, functions) with the correspondingapplication. In response to the touching, manipulation or otherinteractions with the keys, the AR system may cause correspondingapplications to be activated and/or provide corresponding inputs to theapplications.

User Interfaces

Using the principles of gesture tracking and/or totem tracking discussedabove, the AR system is configured to create various types of userinterfaces for the user to interact with. With the AR system, any spacearound the user may be converted into a user interface such that theuser can interact with the system. Thus, the AR system does not requirea physical user interface such as a mouse/keyboard, etc. (althoughtotems may be used as reference points, as described above), but rathera virtual user interface may be created anywhere and in any form to helpthe user interact with the AR system.

In one embodiment, there may be predetermined models or templates ofvarious virtual user interfaces. For example, during set-up the user maydesignate a preferred type or types of virtual UI (e.g., body centricUI, head-centric UI, hand-centric UI, etc.) Or, various applications maybe associated with their own types of virtual UI. Or, the user maycustomize the UI to create one that he/she may be most comfortable with.For example, the user may simply, using a motion of his hands “draw” avirtual UI in space and various applications or functionalities mayautomatically populate the drawn virtual UI.

Before delving into various embodiments of user interfaces, an exemplaryprocess 4100 of interacting with a user interface with be brieflydescribed.

Referring now to flowchart 4100 of FIG. 47, in step 4102, the AR systemmay identify a particular UI. The type of UI may be predetermined by theuser. The system may identify that a particular UI needs to be populatedbased on a user input (e.g, gesture, visual data, audio data, sensorydata, direct command, etc.). In step 4104, the AR system may generatedata for the virtual UI. For example, data associated with the confines,general structure, shape of the UI etc. may be generated. In addition,the AR system may determine map coordinates of the user's physicallocation so that the AR system can display the UI in relation to theuser's physical location.

For example, if the UI is body centric, the AR system may determine thecoordinates of the user's physical stance such that a ring UI can bedisplayed around the user. Or, if the UI is hand centric, the mapcoordinates of the user's hands may need to be determined. It should beappreciated that these map points may be derived through data receivedthrough the FOV cameras, sensory input, or any other type of collecteddata.

In step 4106, the AR system may send the data to the user device fromthe cloud. Or the data may be sent from a local database to the displaycomponents. In step 4108, the UI is displayed to the user based on thesent data.

Once the virtual UI has been created, the AR system may simply wait fora command from the user to generate more virtual content on the virtualUI in step 4110. For example, the UI maybe a body centric ring aroundthe user's body. The AR system may then wait for the command, and if itis recognized (step 4112), virtual content associated with the commandmay be displayed to the user. The following are various examples of userinterfaces that may be created for the user. However the process flowdiagram will be similar to that described above.

FIG. 48A shows a user interacting via gestures with a user interfacevirtual construct rendered by the AR system, according to oneillustrated embodiment.

In particular, FIG. 48A (scene 4810) shows the user interacting with agenerally annular layout or configuration virtual user interface ofvarious user selectable virtual icons. The user selectable virtual iconsmay represent applications (e.g., social media application, Web browser,electronic mail application), functions, menus, virtual rooms or virtualspaces, etc.

The user may, for example, perform a swipe gesture. The AR systemdetects the swipe gesture, and interprets the swipe gesture as aninstruction to render the generally annular layout or configuration userinterface. The AR system then renders the generally annular layout orconfiguration virtual user interface into the user's field of view so asto appear to at least partially surround the user, spaced from the userat a distance that is within arm's reach of the user.

FIG. 48B (scene 4820) shows a user interacting via gestures, accordingto one illustrated embodiment. The generally annular layout orconfiguration virtual user interface may present the various userselectable virtual icons in a scrollable form. The user may gesture, forexample with a sweeping motion of a hand, to cause scrolling throughvarious user selectable virtual icons. For instance, the user may make asweeping motion to the user's left or to the user' right, in order tocause scrolling in the left (i.e., counterclockwise) or right (i.e.,clockwise) directions, respectively.

In particular, FIG. 48B shows the user interacting with the generallyannular layout or configuration virtual user interface of various userselectable virtual icons of FIG. 48A. Identical or similar physicaland/or virtual elements are identified using the same reference numbersas in FIG. 48A, and discussion of such physical and/or virtual elementswill not be repeated in the interest of brevity.

The user may, for example, perform a point or touch gesture, proximallyidentifying one of the user selectable virtual icons. The AR systemdetects the point or touch gesture, and interprets the point or touchgesture as an instruction to open or execute a correspondingapplication, function, menu or virtual room or virtual space. The ARsystem then renders appropriate virtual content based on the userselection.

FIG. 48C (scene 4830) shows the user interacting with the generallyannular layout or configuration virtual user interface of various userselectable virtual icons. Identical or similar physical and/or virtualelements are identified using the same reference numbers as in FIG. 48A,and discussion of such physical and/or virtual elements will not berepeated in the interest of brevity.

In particular, the user selects one of the user selectable virtualicons. In response, the AR system opens or executes a correspondingapplication, function, menu or virtual room or virtual space. Forexample, the AR system may render a virtual user interface for acorresponding application as illustrated in FIG. 48C. Alternatively, theAR system may render a corresponding virtual room or virtual space basedon the user selection.

As discussed above, virtual user interfaces may also be created throughuser gestures. Before delving into various embodiments of creating Uls,FIG. 49 is an exemplary process flow diagram 4300 of creating userinterfaces based on the user's gestures/finger or hand position. In step4302, the AR system detects a movement of the user's fingers or hands.

This movement may be a predetermined gesture signifying that the userwants to create an interface (the AR system may compare the gesture to amap of predetermined gestures, for example). Based on this, the ARsystem may recognize the gesture as a valid gesture in step 4304. Instep 4304, the AR system may retrieve through the cloud server, a set ofmap points associated with the user's position of fingers/hands in orderto display the virtual UI at the right location, and in real-time withthe movement of the user's fingers or hands. In step 4306, the AR systemcreates the UI that mirrors the user's gestures, and displayed the UI inreal-time at the right position using the map points (step 4308).

The AR system may then detect another movement of the fingers hands oranother predetermined gesture indicating to the system that the creationof user interface is done (step 4310). For example the user may stopmaking the motion of his fingers, signifying to the AR system to stop“drawing” the UI. In step 4312, the AR system displays the UI at the mapcoordinates equal to that of the user's fingers/hands when making thegesture indicating to the AR system that the user desires creations of acustomized virtual UI. The following figures go through variousembodiments of virtual UI constructions, all of which may be createdusing similar processes as described above.

FIG. 50A (scene 5002) shows a user interacting via gestures with a userinterface virtual construct rendered by an AR system according to oneillustrated embodiment.

In particular, FIG. 50A shows a user performing a gesture to create anew virtual work portal or construct in hovering in space in a physicalenvironment or hanging or glued to a physical surface such as a wall ofa physical environment. The user may, for example, perform a two armgesture, for instance dragging outward from a center point to locationswhere an upper left and a lower right corner of the virtual work portalor construct should be located. The virtual work portal or constructmay, for example, be represented as a rectangle, the user gestureestablishing not only the position, but also the dimensions of thevirtual work portal or construct.

The virtual work portal or construct may provide access to other virtualcontent, for example to applications, functions, menus, tools, games,and virtual rooms or virtual spaces. The user may employ various othergestures for navigating once the virtual work portal or construct hasbeen created or opened.

FIG. 50B (scene 5004) shows a user interacting via gestures with a userinterface virtual construct rendered by an AR system, according to oneillustrated embodiment.

In particular, FIG. 50B shows a user performing a gesture to create anew virtual work portal or construct on a physical surface of a physicalobject that serves as a totem. The user may, for example, perform a twofinger gesture, for instance an expanding pinch gesture, draggingoutward from a center point to locations where an upper left and a lowerright corner of the virtual work portal or construct should be located.The virtual work portal or construct may, for example, be represented asa rectangle, the user gesture establishing not only the position, butalso the dimensions of the virtual work portal or construct.

The virtual work portal or construct may provide access to other virtualcontent, for example to applications, functions, menus, tools, games,and virtual rooms or virtual spaces. The user may employ various othergestures for navigating once the virtual work portal or construct hasbeen created or opened.

FIG. 50C (scene 5006) shows a user interacting via gestures with a userinterface virtual construct rendered by an AR system, according to oneillustrated embodiment.

In particular, FIG. 50C shows a user performing a gesture to create anew virtual work portal or construct on a physical surface such as a topsurface of a physical table or desk. The user may, for example, performa two arm gesture, for instance dragging outward from a center point tolocations where an upper left and a lower right corner of the virtualwork portal or construct should be located. The virtual work portal orconstruct may, for example, be represented as a rectangle, the usergesture establishing not only the position, but also the dimensions ofthe virtual work portal or construct.

As illustrated in FIG. 50C, specific application, functions, tools,menus, models, or virtual rooms or virtual spaces can be assigned orassociated to specific physical objects or surfaces. Thus, in responseto a gesture performed on or proximate a defined physical structure orphysical surface, the AR system automatically opens the respectiveapplication, functions, tools, menus, model, or virtual room or virtualspace associated with the physical structure or physical surface,eliminating the need to navigate the user interface.

As previously noted, a virtual work portal or construct may provideaccess to other virtual content, for example to applications, functions,menus, tools, games, three-dimensional models, and virtual rooms orvirtual spaces. The user may employ various other gestures fornavigating once the virtual work portal or construct has been created oropened.

FIGS. 51A-51C (scenes 5102-5106) show a user interacting via gestureswith a user interface virtual construct rendered by an AR system (notshown in FIGS. 51A-51C), according to one illustrated embodiment.

The user interface may employ either or both of at least two distincttypes of user interactions, denominated as direct input or proxy input.Direct input corresponds to conventional drag and drop type userinteractions, in which the user selects an iconification of an instanceof virtual content, for example with a pointing device (e.g., mouse,trackball, finger) and drags the selected icon to a target (e.g.,folder, other iconification of for instance an application).

Proxy input corresponds to a user selecting an iconification of aninstance of virtual content by looking or focusing one the specificiconification with the user's eyes, then executing some other action (s)(e.g., gesture), for example via a totem. A further distinct type ofuser input is denominated as a throwing input. Throwing inputcorresponds to a user making a first gesture (e.g., grasping orpinching) to select selects an iconification of an instance of virtualcontent, followed by a second gesture (e.g., arm sweep or throwingmotion towards target) to indicate a command to move the virtual contentat least generally in a direction indicated by the second gesture. Thethrowing input will typically include a third gesture (e.g., release) toindicate a target (e.g., folder, other iconification of for instance anapplication).

The third gesture may be performed when the user's hand is aligned withthe target or at least proximate the target. The third gesture may beperformed when the user's hand is moving in the general direction of thetarget but not yet aligned or proximate with the target, assuming thatthere is no other virtual content proximate the target which wouldrender the intended target ambiguous to the AR system.

Thus, the AR system detects and responds to gestures (e.g., throwinggestures, pointing gestures) which allow freeform location specificationof a location at which virtual content should be rendered or moved. Forexample, where a user desires a virtual display, monitor or screen, theuser may specify a location in the physical environment in the user'sfield of view in which to cause the virtual display, monitor or screento appear.

This contrasts from gesture input to a physical device, where thegesture may cause the physical device to operate (e.g., ON/OFF, changechannel or source of media content), but does not change a location ofthe physical device.

Additionally, where a user desires to logically associate a firstinstance of virtual content (e.g., icon representing file) with a secondinstance (e.g., icon representing storage folder or application), thegesture defines a destination for the first instance of virtual content.

In particular, FIG. 51A shows the user performing a first gesture toselect a virtual content in the form of a virtual work portal orconstruct. The user may for example, perform a pinch gesture, pinchingand appear to hold the virtual work portal or construct between a thumband index finger. In response to the AR system detecting a selection(e.g., grasping, pinching or holding) of a virtual work portal orconstruct, the AR system may re-render the virtual work portal orconstruct with visual emphasis, for example as show in FIG. 88A.

The visual emphasis cues the user at to which piece of virtual contentthe AR system has detected as being selected, allowing the user tocorrect the selection if necessary. Other types of visual cues oremphasis may be employed, for example highlighting, marqueeing,flashing, color changes, etc.

In particular, FIG. 51B shows the user performing a second gesture tomove the virtual work portal or construct to a physical object, forexample a surface of a wall, on which the user wishes to map the virtualwork portal or construct. The user may, for example, perform a sweepingtype gesture while maintaining the pinch gesture. In someimplementations, the AR system may determine which physical object theuser intends, for example based on either proximity and/or a directionof motion.

For instance, where a user makes a sweeping motion toward a singlephysical object, the user may perform the release gesture with theirhand short of the actual location of the physical object. Since thereare no other physical objects in proximate or in line with the sweepinggesture when the release gesture is performed, the AR system canunambiguously determine the identity of the physical object that theuser intended. This may in some ways be thought of as analogous to athrowing motion.

In response to the AR system detecting an apparent target physicalobject, the AR system may render a visual cue positioned in the user'sfield of view so as to appear co-extensive with or at least proximatethe detected intended target. For example, the AR system may render aboarder that encompasses the detected intended target as shown in FIG.49B.

The AR system may also continue render the virtual work portal orconstruct with visual emphasis, for example as shown in FIG. 49B. Thevisual emphasis cues the user as to which physical object or surface theAR system has detected as being selected, allowing the user to correctthe selection if necessary. Other types of visual cues or emphasis maybe employed, for example highlighting, marqueeing, flashing, colorchanges, etc.

In particular, FIG. 51C shows the user performing a third gesture toindicate a command to map the virtual work portal or construct to theidentified physical object, for example a surface of a wall, to causethe AR system to map the virtual work portal or construct to thephysical object. The user may, for example, perform a release gesture,releasing the pinch to simulate releasing the virtual work portal orconstruct.

FIGS. 52A-52C (scenes 5202-5206) show a number of user interface virtualconstructs rendered by an AR system (not shown in FIGS. 52A-52C) inwhich a user's hand serves as a totem, according to one illustratedembodiment. It should be appreciated that FIGS. 52A-C may follow theprocess flow diagram of FIG. 47 in order to create a user interface onthe user's hands.

As illustrated in FIG. 52A, in response to detecting a first definedgesture (e.g., user opening or displaying open palm of hand, userholding up hand), the AR system renders a primary navigation menu in afield of view of the user so as to appear to be on or attached to aportion of the user's hand. For instance, a high level navigation menuitem, icon or field may be rendered to appear on each finger other thanthe thumb. The thumb may be left free to serve as a pointer, whichallows the user to select a desired one of the high level navigationmenu item or icons via one of second defined gestures, for example bytouch the thumb to the corresponding fingertip.

The menu items, icons or fields may, for example, represent userselectable virtual content, for instance applications, functions, menus,tools, models, games, and virtual rooms or virtual spaces.

As illustrated in FIG. 52B, in response to detecting a third definedgesture (e.g., user spreads fingers apart), the AR system expands themenus, rendering an a lower level navigation menu in a field of view ofthe user so as to appear to be on or attached to a portion of the user'shand. For instance, a number of lower level navigation menu items oricons may be rendered to appear on each of the fingers other than thethumb. Again, the thumb may be left free to serve as a pointer, whichallows the user to select a desired one of the lower level navigationmenu item or icons by touch the thumb to a corresponding portion of thecorresponding finger.

As illustrated in FIG. 52C, in response to detecting a fourth definedgesture (e.g., user making circling motion in palm of hand with fingerfrom other hand), the AR system scrolls through the menu, renderingfields of the navigation menu in a field of view of the user so as toappear to be on or attached to a portion of the user's hand. Forinstance, a number of fields may appear to scroll successively from onefinger to the next. New fields may scroll into the field of view,entering form one direction (e.g., from proximate the thumb) and otherfields may scroll from the field of view, existing from the otherdirection (e.g., proximate the pinkie finger). The direction ofscrolling may correspond to a rotational direction of the finger in thepalm. For example the fields may scroll in one direction in response toa clockwise rotation gesture and scroll in a second, opposite direction,in response to a counterclockwise rotation gesture.

User Scenarios—Interacting with Passable World Model and/or MultipleUsers

Using the principles of gesture tracking/UI creation, etc. a fewexemplary user applications will now be described. The applicationsdescribed below may have hardware and/or software components that may beseparate installed onto the system, in some embodiments. In otherembodiments, the system may be used in various industries, etc. and maybe modified to achieve some of the embodiments below.

Prior to delving into specific applications or user scenarios, anexemplary process of receiving and updating information from thepassable world model will be briefly discussed. The passable worldmodel, discussed above, allows multiple users to access the virtualworld stored on a cloud server and essentially pass on a piece of theirworld to other users. For example, similar to other examples discussedabove, a first user of an AR system in London may want to conference inwith a second user of the AR system currently located in New York.

The passable world model may enable the first user to pass on a piece ofthe passable world that constitutes the current physical surroundings ofthe first user to the second user, and similarly pass on a piece of thepassable world that constitutes an avatar of the second user such thatthe second user appears to be in the same room as the first user inLondon. In other words, the passable world allows the first user totransmit information about the room to the second user, andsimultaneously allows the second user to create an avatar to placehimself in the physical environment of the first user. Thus, both usersare continuously updating, transmitting and receiving information fromthe cloud, giving both users the experience of being in the same room atthe same time.

Referring to FIG. 53, an exemplary process 5300 of how data iscommunicated back and forth between two users located at two separatephysical locations is disclosed. It should be appreciated that eachinput system (e.g., sensors, cameras, eye tracking, audio, etc.) mayhave a process similar to the one below. For illustrative purposes, theinput of the following system may be input from the FOV cameras (e.g.,cameras that capture the FOV of the users).

In step 3402, the AR system may check for input from the cameras. Forexample, following the above example, the user in London may be in aconference room, and may be drawing some figures on the white board.This may or may not constitute input for the AR system. Since thepassable world is constantly being updated and built upon data receivedfrom multiple users, the virtual world existing on the cloud becomesincreasingly precise, such that only new information needs to be updatedto the cloud.

For example, if the user simply moved around the room, there may alreadyhave been enough 3D points, pose data information, etc. such that theuser device of the user in New York is able to project the conferenceroom in London without actively receiving new data from the user inLondon. However, if the user in London is adding new information, suchas drawing a figure on the board in the conference room, this mayconstitute input that needs to be transmitted to the passable worldmodel, and passed over to the user in New York. Thus, in step 3404, theuser device checks to see if the received input is valid input. If thereceived input is not valid, there is wait loop in place such that thesystem simply checks for more input 3402

If the input is valid, the received input is fed to the cloud server instep 3406. For example, only the updates to the board may be sent to theserver, rather than sending data associated with all the pointscollected through the FOV camera.

On the cloud server, in step 3408, the input is received from the userdevice, and updated into the passable world model in step 3410. Asdiscussed in other system architectures described above, the passableworld model on the cloud server may have processing circuitry, multipledatabases, including a mapping database with both geometric andtopological maps, object recognizers and other suitable softwarecomponents.

In step 3410, based on the received input 3408, the passable world modelis updated. The updates may then be sent to various user devices thatmay need the updated information, in step 3412. Here, the updatedinformation may be sent to the user in New York such that the passableworld that is passed over to the user in New York can also view thefirst user's drawing as a picture is drawn on the board in theconference room in London.

It should be appreciated that the second user's device may already beprojecting a version of the conference room in London, based on existinginformation in the passable world model, such that the second user inNew York perceives being in the conference room in London. In step 3426,the second user device receives the update from the cloud server. Instep 3428, the second user device may determine if the update needs tobe displayed. For example, certain changes to the passable world may notbe relevant to the second user and may not be updated. In step 3430, theupdated passable world model is displayed on the second user's hardwaredevice. It should be appreciated that this process of sending andreceiving information from the cloud server is performed rapidly suchthat the second user can see the first user drawing the figure on theboard of the conference room almost as soon as the first user performsthe action.

Similarly, input from the second user is also received in steps3420-3424, and sent to the cloud server and updated to the passableworld model. This information may then be sent to the first user'sdevice in steps 3414-3418. For example, assuming the second user'savatar appears to be sitting in the physical space of the conferenceroom in London, any changes to the second user's avatar (which may ormay not mirror the second user's actions/appearance) must also betransmitted to the first user, such that the first user is able tointeract with the second user.

In one example, the second user may create a virtual avatar resemblinghimself, or the avatar may be a bee that hovers around the conferenceroom in London. In either case, inputs from the second user (forexample, the second user may shake his head in response to the drawingsof the first user), are also transmitted to the first user such that thefirst user can gauge the second user's reaction. In this case, thereceived input may be based on facial recognition and changes to thesecond user's face may be sent to the passable world model, and thenpassed over to the first user's device such that the change to theavatar being projected in the conference room in London is seen by thefirst user.

Similarly, there may be many other types of input that are effectivelypassed back and forth between multiple users of the AR system. Althoughthe particular examples may change, all interactions between a user ofthe AR system and the passable world is similar to the process describedabove, with reference to FIG. 53. While the above process flow diagramdescribes interaction between multiple users accessing and passing apiece of the passable world to each other, FIG. 54 is an exemplaryprocess flow diagram 4400 illustrating interaction between a single userand the AR system. The user may access and interact with variousapplications that require data retrieved from the cloud server.

In step 4402, the AR system checks for input from the user. For example,the input may be visual, audio, sensory input, etc. indicating that theuser requires data. For example, the user may want to look upinformation about an advertisement he may have just seen on a virtualtelevision. In step 4404, the system determines if the user input isvalid. If the user input is valid, in step 4406, the input is fed intothe server. On the server side, when the user input is received in step4408, appropriate data is retrieved from a knowledge base in step 4410.As discussed above, there may be multiple knowledge databases connectedto the cloud server from which to retrieve data. In step 4412, the datais retrieved and transmitted to the user device requesting data.

Back on the user device, the data is received from the cloud server instep 4414. In step 4416, the system determines when the data needs to bedisplayed in the form of virtual content, and if it does, the data isdisplayed on the user hardware 4418.

As discussed briefly above, many user scenarios may involve the ARsystem identifying real-world activities and automatically performingactions and/or displaying virtual content based on the detectedreal-world activity. For example, the AR system recognizes the useractivity (e.g., cooking) and then creates a user interface that floatsaround the user's frame of reference providing usefulinformation/virtual content associated with the activity. Similarly,many other uses can be envisioned, some of which will be described inuser scenarios below.

Referring now to FIG. 55, an exemplary process flow diagram 4200 ofrecognizing real-world activities will be briefly described. In step4202, the AR system may receive data corresponding to a real-worldactivity. For example, the data may be visual data, audio data, sensorydata, etc. Based on the received data, the AR system may identify thereal-world activity in step 4204.

For example, the captured image of a user cutting vegetables may berecorded, and when compared to a mapping database, the AR system mayrecognize that the user is cooking, for example. Based on the identifiedreal-world activity, the AR system may load a knowledge base associatedwith the real-world activity in step 4206, using the process flowdiagram of FIG. 54, for example. Or, the knowledge base may be a locallystored knowledge base.

Once the knowledge base has been loaded, the AR system may rely onspecific activities within the broad category to determine usefulinformation to be displayed to the user. For example, the AR system mayhave retrieved information related to cooking, but may only need todisplay information about a particular recipe that the user is currentlymaking. Or the AR system may only need to display information aboutcooking which is determined based on receiving further input from theuser, in step 4208.

The AR system may then determine the specific activity in step 4210,similar to step 4202-4204, based on the received input regarding thespecific activity. In step 4212, the AR system may check the loadedknowledge base to determine relevant data associated with the specificactivity and display the relevant information/virtual content in theuser interface (e.g., floating user interface). In step 4216, the ARsystem determines whether further user feedback is received. In steps4218 and 4220, the user either performs an action based on user feedbackor simply waits for further feedback related to the real-world activity.The following user scenarios may use one or more of the process flowdiagrams outlined above.

FIG. 56A shows a user sitting in a physical office space, and using anAR system to experience a virtual room or virtual space in the form of avirtual office, at a first time, according to one illustratedembodiment.

The physical office may include one or more physical objects, forinstance walls, floor (not shown), ceiling (not shown), a desk andchair. The user may wear a head worn AR system, or head worn componentof an AR system. The head worn AR system or component is operable torender virtual content in a field of view of the user. For example, thehead worn AR system or component may render virtual objects, virtualtools and applications onto the retina of each eye of the user.

As illustrated the AR system renders a virtual room or virtual space inthe form of a virtual office, in which the user performs theiroccupation or job. Hence, the virtual office is populated with variousvirtual tools or applications useful in performing the user's job. Thevirtual tools or applications may for example include various virtualobjects or other virtual content, for instance two-dimensional drawingsor schematics, two-dimensional images or photographs, and athree-dimensional architectural model.

The virtual tools or applications may for example include tools such asa ruler, caliper, compass, protractor, templates or stencils, etc. Thevirtual tools or applications may for example include interfaces forvarious software applications, for example interfaces for email, a Webbrowser, word processor software, presentation software, spreadsheetsoftware, voicemail software, etc. Some of the virtual objects may bestacked or overlaid with respect to one another. The user may select adesired virtual object with a corresponding gesture.

Based on the recognized gesture, the AR system may map the gesture, andrecognize the command. The command may be to move the user interface,and may then display the next virtual object. For instance, the user maypage through documents or images with a finger flicking gesture toiteratively move through the stack of virtual objects. Some of thevirtual objects may take the form of menus, selection of which may causerendering of a submenu. The user scenario illustrated in FIGS. 56A-56B(scenes 5602 and 5604) may utilize aspects of the process flow diagramsillustrated in FIGS. 54 and 55.

FIG. 56B shows the user in the physical office employing the virtualoffice of FIG. 56A, at a second time, according to one illustratedembodiment. The physical office of FIG. 56B is identical to that of FIG.56A, and the virtual office of FIG. 56B is similar to the virtual officeof FIG. 56A.

At the second time, the AR system may present (i.e., render) a virtualalert or notification to the user in the virtual office. The virtualalert may be based on data retrieved from the cloud. Or for example, thevirtual alert may be based on identifying a real-world activity, asdescribed in FIG. 55. For example, the AR system may render a visualrepresentation of a virtual alert or notification in the user's field ofview. The AR system may additionally or alternatively render an auralrepresentation of a virtual alert or notification.

FIG. 57 (scene 5700) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual office, at a first time, according to oneillustrated embodiment.

The physical living room may include one or more physical objects, forinstance walls, floor, ceiling, a coffee table and sofa. The user maywear a head worn AR system, or head worn component of an AR system. Thehead worn AR system or component is operable to render virtual contentin a field of view of the user. For example, the head worn AR system orcomponent may render virtual objects, virtual tools and applicationsonto the retina of each eye of the user.

As illustrated the AR system renders a virtual room or virtual space inthe form of a virtual office, in which the user performs theiroccupation or job. Hence, the virtual office is populated with variousvirtual tools or applications useful in performing the user's job. Thismay be based on received inputs by the user, based on which the ARsystem may retrieve data from the cloud and display the virtual tools tothe user.

As FIGS. 56A and 57 illustrate, a virtual office may be portable, beingrenderable in various different physical environments. It thus may beparticularly advantageous if the virtual office renders identically in asubsequent use to its appearance or layout as the virtual officeappeared in in a most previous use or rendering. Thus, in eachsubsequent use or rendering, the same virtual objects will appear andthe various virtual objects may retain their same spatial positionsrelative to one another as in a most recently previous rendering of thevirtual office.

In some implementations, this consistency or persistence of appearanceor layout from one use to next subsequent use, may be independent of thephysical environments in which the virtual space is rendered. Thus,moving from a first physical environment (e.g., physical office space)to a second physical environment (e.g., physical living room) will notaffect an appearance or layout of the virtual office.

The user may, for example select a specific application (e.g., cameraapplication), for use while in a specific virtual room or virtual space(e.g., office space).

FIG. 58 (scene 5800) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual entertainment or media room, at a firsttime, according to one illustrated embodiment.

The user may wear a head worn AR system, or head worn component of an ARsystem. The head worn AR system or component is operable to rendervirtual content in a field of view of the user. For example, the headworn AR system or component may render virtual objects, virtual toolsand applications onto the retina of each eye of the user.

As illustrated the AR system renders a virtual room or virtual space inthe form of a virtual entertainment or media room, in which the userrelaxes and/or enjoys entertainment or consumes media (e.g., programs,movies, games, music, reading). Hence, the virtual entertainment ormedia room is populated with various virtual tools or applicationsuseful in enjoying entertainment and/or consuming media.

The AR system may render the virtual entertainment or media room with avirtual television or primary screen. Since the AR system may rendervirtual content to a user's retina, the virtual television or primaryscreen can be rendered to any desired size. The virtual television orprimary screen could even extend beyond the confines of the physicalroom. The AR system may render the virtual television or primary screento replicate any know or yet to be invented physical television.

Thus, the AR system may render the virtual television or primary screento replicate a period or classic television from the 1950s, 1960, or1970s, or may replicate any current television. For example, the virtualtelevision or primary screen may be rendered with an outward appears ofa specific make and model and year of a physical television. Also forexample, the virtual television or primary screen may be rendered withthe same picture characteristics of a specific make and model and yearof a physical television. Likewise, the AR system may render sound tohave the same aural characteristics as sound from a specific make andmodel and year of a physical television.

The AR system also renders media content to appear as if the mediacontent was being displayed by the virtual television or primary screen.The AR system may retrieve data from the cloud, such that virtualtelevision displaying virtual content that is streamed from the passableworld model, based on received user input indicating that the user wantsto watch virtual television. Here, the user may also create the userinterface, to specify the confines of the user interface or virtualtelevision, similar to the process flow diagram of FIG. 49 discussedabove. The media content may take any of a large variety for forms,including television programs, movies, video conference or calls, etc.

The AR system may render the virtual entertainment or media room withone or more additional virtual televisions or secondary screens.Additional virtual televisions or secondary screens may enable the userto enjoy second screen experiences.

For instance, a first secondary screen may allow the user to monitor astatus of a fantasy team or player in a fantasy league (e.g., fantasyfootball league), including various statistics for players and teams.Again, based on user input received from the user regarding the type ofvirtual content desired and a location of the virtual content, the ARsystem may retrieve data from the cloud server and display it at thelocation desired by the user, as per process flow diagrams of FIGS. 49,54 and 55.

Additionally or alternatively, a second or more secondary screens mayallow the user to monitor other activities, for example activitiestangentially related to the media content on the primary screen. Forinstance, a second or additional secondary screens may display a listingof scores in games from around a conference or league while the userwatches one of the games on the primary screen.

Also for instance, a second or additional secondary screens may displayhighlights from games from around a conference or league, while the userwatches one of the games on the primary screen. One or more of thesecondary screens may be stacked as illustrated FIG. 30, allowing a userto select a secondary screen to bring to a top, for example via agesture. For instance, the user may use a gesture to toggle through thestack of secondary screens in order, or may use a gesture to select aparticular secondary screen to bring to a foreground relative to theother secondary screens.

The AR system may render the virtual entertainment or media room withone or more three-dimensional replay or playback tablets. Thethree-dimensional replay or playback tablets may replicate in miniature,a pitch or playing field of a game the user is watching on the primarydisplay, for instance providing a “God's eye view.” Thethree-dimensional replay or playback tablets may, for instance, allowthe user to enjoy on-demand playback or replay of media content thatappears on the primary screen. This may include user selection ofportions of the media content to be play backed or replayed.

This may include user selection of special effects, for example slowmotion replay, stopping or freezing replay, or speeding up or fastmotion replay to be faster than actual time. Such may additionally allowa user to add or introduce annotations into the display. For example,the user may gesture to add annotations marking a receiver's routeduring a replay of a play in a football game, or to mark a blockingassignment for a linemen or back.

The three-dimensional replay or playback tablet may even allow a user toadd a variation (e.g., different call) that modifies how a previous playbeing reviewed plays out. For example, the user may specify a variationin a route run by a receiver, or a blocking assignment assigned to alineman or back. The AR system may use the fundamentals parameters ofthe actual play, modifying one or more parameters, and then executing agame engine on the parameters to play out a previous play executed in anactual physical game but with the user modification(s). For example, theuser may track an alternative route for a wide receiver. The AR systemhas all makes no changes to the actions of the players, except theselected wide receiver, the quarterback, and any defensive players whowould cover the wide receiver.

An entire virtual fantasy play may be played out, which may even producea different outcome than the actual play. This may occur, for example,during an advertising break or time out during the game. This allows theuser to test their abilities as an armchair coach or player. A similarapproach could be applied to other sports. For example, the user maymake a different play call in a replay of a basketball game, or may callfor a different pitch in a replay of a baseball game, to name just a fewexamples. Use of a game engine allows the AR system to introduce anelement of statistical chance, but within the confines of what would beexpected in real games.

The AR system may render additional virtual content, for example 3Dvirtual advertisements. The subject matter or content of the 3D virtualadvertisements may, for example, be based at least in part on thecontent of what is being played or watched on the virtual television orprimary screen. The AR system may detect a real-world activity and thenautomatically display virtual content based on the virtual contentsimilar to the process flow described in FIG. 55 above.

The AR system may render virtual controls. For example, the AR systemmay render virtual controls mapped in the user's field of vision so asto appear to be within arm's reach of the user. The AR system maymonitor of user gestures toward or interaction with the virtualcontrols, and cause corresponding actions in response to the gestures orinteractions.

The AR system allows users to select a virtual room or space to berendered to the user's field of view, for example as a 4D light field.For example, the AR system may include a catalog or library of virtualrooms or virtual spaces to select from. The AR system may include ageneric or system wide catalog or library of virtual rooms or virtualspaces, which are available to all users. The AR system may include anentity specific catalog or library of virtual rooms or virtual spaces,which are available to a subset of users, for example users who are allaffiliated with a specific entity such as a business, institution orother organization. The AR system may include a number of user specificcatalogs or libraries of virtual rooms or virtual spaces, which areavailable to respective specific users or others who are authorized orgranted access or permission by the respective specific user.

The AR system allows users to navigate from virtual space to virtualspace. For example, a user may navigate between a virtual office spaceand a virtual entertainment or media space. As discussed herein, the ARsystem may be responsive to certain user input to allow navigationdirectly from one virtual space to another virtual space, or to toggleor browse through a set of available virtual spaces. The set of virtualspaces may be specific to a user, specific to an entity to which a userbelongs, and/or may be system wide or generic to all users.

To allow user selection of and/or navigation between virtual rooms orvirtual spaces, the AR system may be responsive to one or more of, forinstance, gestures, voice commands, eye tracking, and/or selection ofphysical buttons, keys or switches for example carried by a head worncomponent, belt pack or other physical structure of the individual ARsystem. The user input may be indicative of a direct selection of avirtual space or room, or may cause a rendering of a menu or submenus toallow user selection of a virtual space or room.

FIG. 59 (scene 5900) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual entertainment or media room, at a firsttime, according to one illustrated embodiment.

The physical living room may include one or more physical objects, forinstance walls, floor, ceiling, a coffee table and sofa. As previouslynoted, the user may wear a head worn AR system, or head worn componentof an AR system, operable to render virtual content in a field of viewof the user. For example, the head worn AR system or component mayrender virtual objects, virtual tools and applications onto the retinaof each eye of the user.

The AR system may store a set of virtual rooms or spaces that arelogically associated with a specific physical location, physical room orphysical space. For example, the AR system may store a mapping between aphysical location, physical room or physical space and one or morevirtual rooms or spaces. For instance, the AR system may store a mappingbetween a user's physical living room and a virtual entertainment ormedia room.

Also for instance, the AR system may store a mapping between the user'sphysical living room and a number of other virtual rooms or spaces(e.g., office space). The AR system may determine a current location ofa user, and detect a specific user gesture (single headed arrow). Basedon knowledge of the user's current physical location, and in response tothe gesture, the AR system may render virtual content that scrolls ortoggles through the set of virtual rooms or virtual spaces mapped orotherwise associated with the specific physical space. For example, theAR system may render the virtual content associated with a next one ofthe virtual rooms or spaces in a set

As illustrated in FIG. 59, the AR system may render a user interfacetool which provides a user with a representation of choices of virtualrooms or virtual spaces, and possibly a position of a currently selectedvirtual room or virtual space in a set of virtual room or virtual spaceavailable to the user. As illustrated, the representation takes the formof a line of marks or symbols, with each marking representing arespective one of the virtual rooms or virtual spaces available to theuser. A currently selected one of the virtual rooms or virtual spaces isvisually emphasized, to assist the user in navigating forward orbackward through the set.

FIGS. 60A, 60B (scenes 6002 and 6004) show a user sitting in a physicalliving room space, and using an AR system to experience a virtual roomor virtual space in the form of a virtual entertainment or media room,the user executing gestures to interact with a user interface virtualconstruct, according to one illustrated embodiment.

The physical living room may include one or more physical objects, forinstance walls, floor, ceiling, a coffee table and sofa. As previouslynoted, the user may wear a head worn AR system, or head worn componentof an AR system, operable to render virtual content in a field of viewof the user. For example, the head worn AR system or component mayrender virtual objects, virtual tools and applications onto the retinaof each eye of the user.

As illustrated in FIG. 60A, the user executes a first gesture(illustrated by double headed arrow), to open an icon based cluster userinterface virtual construct (FIG. 60B). The gesture may include movementof the user's arms and/or hands or other parts of the user's body, forinstance head pose or eyes. Alternatively, the user may use spokencommands to access the icon based cluster user interface virtualconstruct (FIG. 60B). If a more comprehensive menu is desired, the usermay use a different gesture.

As illustrated in FIG. 60B, the icon based cluster user interfacevirtual construct provides a set of small virtual representations of avariety of different virtual rooms or spaces from which a user mayselect. This virtual user interface provides quick access to virtualrooms or virtual spaces via representations of the virtual rooms orvirtual spaces. The small virtual representations are themselvesessentially non-functional, in that they do not include functionalvirtual content. Thus, the small virtual representations arenon-functional beyond being able to cause a rendering of a functionalrepresentation of a corresponding virtual room or space in response toselection of one of the small virtual representations.

The set of small virtual representations may correspond to a set orlibrary of virtual rooms or spaces available to the particular user.Where the set includes a relatively large number of choices, the iconbased cluster user interface virtual construct may, for example, allow auser to scroll through the choice. For example, in response to a secondgesture, an AR system may re-render the icon based cluster userinterface virtual construct with the icons shifted in a first direction(e.g., toward user's right), with one icon falling out of a field ofview (e.g., right-most icon) and a new icon entering the field of view.The new icon corresponds to a respective virtual room or virtual spacethat was not displayed, rendered or shown in a temporally mostimmediately preceding rendering of the icon based cluster user interfacevirtual construct. A third gesture may, for example, cause the AR systemto scroll the icons in the opposite direction (e.g., toward user's left)similar to process flow diagram of FIG. 37).

In response to a user selection of a virtual room or virtual space, theAR system may render virtual content associated with the virtual room orvirtual space to appear in the user's field of view. The virtual contentmay be mapped or “glued” to the physical space. For example, the ARsystem may render some or all of the virtual content positioned in theuser's field of view to appear as if the respective items or instancesof virtual content are on various physical surfaces in the physicalspace, for instance walls, tables, etc. Also for example, the AR systemmay render some or all of the virtual content positioned in the user'sfield of view to appear as if the respective items or instances ofvirtual content are floating in the physical space, for instance withinreach of the user.

FIG. 61A shows a user sitting in a physical living room space, and usingan AR system to experience a virtual room or virtual space in the formof a virtual entertainment or media room, the user executing gestures tointeract with a user interface virtual construct, according to oneillustrated embodiment.

The physical living room may include one or more physical objects, forinstance walls, floor, ceiling, a coffee table and sofa. As previouslynoted, the user may wear a head worn AR system, or head worn componentof an AR system, operable to render virtual content in a field of viewof the user. For example, the head worn AR system or component mayrender virtual objects, virtual tools and applications onto the retinaof each eye of the user.

As illustrated in FIG. 61A (scene 6102), the AR system may render afunctional group or pod user interface virtual construct, so at toappear in a user's field of view, preferably appearing to reside withina reach of the user. The pod user interface virtual construct includes aplurality of virtual room or virtual space based applications, whichconveniently provides access from one virtual room or virtual space tofunctional tools and applications which are logically associated withanother virtual room or virtual space. The pod user interface virtualconstruct forms a mini work station for the user.

As previously discussed, the AR system may render virtual content at anyapparent or perceived depth in the virtual space. Hence, the virtualcontent may be rendered to appear or seem to appear at any depth in thephysical space onto which the virtual space is mapped. Implementation ofintelligent depth placement of various elements or instances of virtualcontent may advantageously prevent clutter in the user's field of view.

As previously noted, the AR system may render virtual content so as toappear to be mounted or glued to a physical surface in the physicalspace, or may render the virtual content so as to appear to be floatingin the physical space. Thus, the AR system may render the pod userinterface virtual construct floating within the reach of the user, whileconcurrently rendering a virtual room or space (e.g., virtualentertainment or media room or space) spaced farther away for the user,for instance appear to be glued to the walls and table.

The AR system detects user interactions with the pod user interfacevirtual construct or the virtual content of the virtual room or space.For example, the AR system may detect swipe gestures, for navigatingthrough context specific rooms. The AR system may render a notificationor dialog box, for example, indicating that the user is in a differentroom. The notification or dialog box may query the use with respect towhat action that the user would like the AR system to take (e.g., closeexisting room and automatically map contents of room, automatically mapcontents of room to existing room, or cancel).

FIG. 61B (scene 6104) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual entertainment or media room, the userexecuting gestures to interact with a user interface virtual construct,according to one illustrated embodiment.

The physical living room may include one or more physical objects, forinstance walls, floor, ceiling, a coffee table and sofa. As previouslynoted, the user may wear a head worn AR system, or head worn componentof an AR system, operable to render virtual content in a field of viewof the user. For example, the head worn AR system or component mayrender virtual objects, virtual tools and applications onto the retinaof each eye of the user.

As illustrated in FIG. 61B, the AR system may render a functional groupor pod user interface virtual construct, so at to appear in a user'sfield of view, preferably appearing to reside within a reach of theuser. The pod user interface virtual construct includes a plurality ofuser selectable representations of virtual room or virtual space basedapplications, which conveniently provides access from one virtual roomor virtual space to functional tools and applications which arelogically associated with another virtual room or virtual space. The poduser interface virtual construct forms a mini work station for the user.This interface allows a user to conveniently navigate existing virtualrooms or virtual spaces to find specific applications, without having tonecessarily render full-scale versions of the virtual rooms or virtualspaces along with the fully functional virtual content that goes alongwith the full-scale versions.

As illustrated in FIG. 61B, the AR system detects user interactions withthe pod user interface virtual construct or the virtual content of thevirtual room or space. For example, the AR system may detect a swipe orpinch gesture, for navigating to and opening context specific virtualrooms or virtual spaces. The AR system may render a visual effect toindicate which of the representations is selected.

FIG. 61C (scene 6106) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual entertainment or media room, the userexecuting gestures to interact with a user interface virtual construct,according to one illustrated embodiment.

As illustrated in FIG. 61C, the AR system may render a selectedapplication in the field of view of the user, in response to a selectionof a representation, such as the selection illustrated in FIG. 61B. Inparticular, the AR system may render a fully functional version of theselected application to the retina of the eyes of the user, for exampleso as to appear on a physical surface (e.g., wall) of the physical roomor physical space (e.g., living room). Notably, the selected applicationan application normally logically associated with another virtual roomor virtual space than the virtual room or virtual space which the useris experiencing. For example, the user may select a social networkingapplication, a Web browsing application, or an electronic mail (email)application from, for example, a virtual work space, while viewing avirtual entertainment or media room or space. Based on this selection,the AR system may retrieve data associated with the application from thecloud server and transmit to the local device, and then may display theretrieved data in the form of the web browsing application, electronicmail, etc. (Similar to process flow of FIG. 54).

FIG. 61D (scene 6108) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual entertainment or media room, the userexecuting gestures to interact with a user interface virtual construct,according to one illustrated embodiment.

The physical living room may include one or more physical objects, forinstance walls, floor, ceiling, a coffee table and sofa. As previouslynoted, the user may wear a head worn AR system, or head worn componentof an AR system, operable to render virtual content in a field of viewof the user. For example, the head worn AR system or component mayrender virtual objects, virtual tools and applications onto the retinaof each eye of the user.

As illustrated in FIG. 61D, the user may perform a defined gesture,which serves as a hot key for a commonly used application (e.g., cameraapplication). The AR system detects the user's gesture, interprets thegesture, and opens or executes the corresponding application. Forexample, the AR system may render the selected application or a userinterface of the selected application in the field of view of the user,in response to the defined gesture. In particular, the AR system mayrender a fully functional version of the selected application orapplication user interface to the retina of the eyes of the user, forexample so as to appear with arm's reach of the user.

A camera application may include a user interface that allows the userto cause the AR system to capture images or image data. For example, thecamera application may allow the user to cause outward facing cameras ona body or head worn component of an individual AR system to captureimages or image data (e.g., 4D light field) of a scene that is in afield of view of the outward facing camera(s) and/or the user.

Defined gestures are preferably intuitive. For example, an intuitive twohanded pinch type gesture for opening a camera application or camerauser interface is illustrated in FIG. 61D. The AR system may recognizeother types of gestures. The AR system may store a catalog or library ofgestures, which maps gestures to respective applications and/orfunctions. Gestures may be defined for all commonly used applications.The catalog or library of gestures may be specific to a particular user.Alternatively or additionally, the catalog or library of gestures may bespecific to a specific virtual room or virtual space. Alternatively, thecatalog or library of gestures may be specific to a specific physicalroom or physical space. Alternatively or additionally, the catalog orlibrary of gestures may be generic across a large number of users and/ora number of virtual rooms or virtual spaces.

As noted above, gestures are preferably intuitive, particular withrelation to the particular function, application or virtual content towhich the respective gesture is logically associated or mapped.Additionally, gestures should be ergonomic. That is the gestures shouldbe comfortable to be performed by users of a wide variety of body sizesand abilities. Gestures also preferably involve a fluid motion, forinstance an arm sweep. Defined gestures are preferably scalable. The setof defined gestures may further include gestures which may be discretelyperformed, particular where discreetness would be desirable orappropriate. On the other hand, some defined gestures should not bediscrete, but rather should be demonstrative, for example gesturesindicating that a user intends to capture images and/or audio of otherspresent in an environment. Gestures should also be culturallyacceptable, for example over a large range of cultures. For instance,certain gestures which are considered offensive in one or more culturesshould be avoided.

A number of proposed gestures are set out in Table A, below.

TABLE A Swipe to the side (Slow) Spread hands apart Bring hands togetherSmall wrist movements (as opposed to large arm movements) Touch body ina specific place (arm, hand, etc.) Wave Pull hand back Swipe to the side(slow) Push forward Flip hand over Close hand Swipe to the side (Fast)Pinch- thumb to forefinger Pause (hand, finger, etc.) Stab (Point)

FIG. 61E (scene 6110) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual entertainment or media room, the userexecuting gestures to interact with a user interface virtual construct,according to one illustrated embodiment.

As illustrated in FIG. 61E, the AR system renders a comprehensivevirtual dashboard menu user interface, for example rendering images tothe retina of the user's eyes. The virtual dashboard menu user interfacemay have a generally annular layout or configuration, at least partiallysurrounding the user, with various user selectable virtual icons spacedto be within arm's reach of the user.

The AR system detects the user's gesture or interaction with the userselectable virtual icons of the virtual dashboard menu user interface,interprets the gesture, and opens or executes a correspondingapplication. For example, the AR system may render the selectedapplication or a user interface of the selected application in the fieldof view of the user, in response to the defined gesture. For example,the AR system may render a fully functional version of the selectedapplication or application user interface to the retina of the eyes ofthe user.

As illustrated in FIG. 61E, the AR system may render media content wherethe application is a source of media content (e.g., ESPN Sports Center®,Netflix®). The AR system may render the application, application userinterface or media content to overlie other virtual content. Forexample, the AR system may render the application, application userinterface or media content to overlie a display of primary content on avirtual primary screen being displayed in the virtual room or space(e.g., virtual entertainment or media room or space).

FIG. 62A (scene 6202) shows a user sitting in a physical living roomspace, and using an AR system to experience a first virtual décor (i.e.,aesthetic skin or aesthetic treatment), the user executing gestures tointeract with a user interface virtual construct, according to oneillustrated embodiment.

The AR system allows a user to change (i.e., re-skin) a virtual décor ofa physical room or physical space. For example, as illustrated in FIG.65A, a user may gesture to bring up a first virtual décor, for example avirtual fireplace with a virtual fire and first and second virtualpictures.

The first virtual décor (e.g., first skin) is mapped to the physicalstructures of the physical room or space (e.g., physical living room).Based on the gesture, the AR system (similar to process flow of FIG. 54)retrieves data associated with the virtual décor and transmits back tothe user device. The retrieved data is then displayed based on the mapcoordinates of the physical room or space.

As also illustrated in FIG. 62A, the AR system may render a userinterface tool which provides a user with a representation of choices ofvirtual décor, and possibly a position of a currently selected virtualdécor in a set of virtual décor available to the user. As illustrated,the representation takes the form of a line of marks or symbols, witheach marking representing a respective one of the virtual décoravailable to the user. A currently selected one of the virtual décor isvisually emphasized, to assist the user in navigating forward orbackward through the set. The set of virtual décor may be specific tothe user, specific to a physical room or physical space, or may beshared by two or more users.

FIG. 62B (scene 6204) shows a user sitting in a physical living roomspace, and using an AR system to experience a second virtual décor(i.e., aesthetic skin or aesthetic treatment), the user executinggestures to interact with a user interface virtual construct, accordingto one illustrated embodiment.

As illustrated in FIG. 62B, a user may gesture to bring up a secondvirtual décor, different from the first virtual décor. The secondvirtual décor may, for example, replicate a command deck of a spacecraft(e.g., Starship) with a view of a planet, technical drawings orillustrations of the spacecraft, and a virtual lighting fixture orluminaire. The gesture to bring up the second virtual décor may beidentical to the gesture to bring up the first virtual décor, the useressentially toggling, stepping or scrolling through a set of definedvirtual decors for the physical room or physical space (e.g., physicalliving room). Alternatively, each virtual décor may be associated with arespective gesture.

As illustrated, a user interface tool may indicate that which of the setof virtual decors is currently selected and mapped to the physical roomor space.

FIG. 62C (scene 6206) shows a user sitting in a physical living roomspace, and using an AR system to experience a third virtual décor (i.e.,aesthetic skin or aesthetic treatment), the user executing gestures tointeract with a user interface virtual construct, according to oneillustrated embodiment.

The physical living room is illustrated as being identical to that ofFIG. 62A. As previously noted, the user may wear a head worn AR system,or head worn component of an AR system, operable to render virtualcontent in a field of view of the user. Identical or similar physicaland/or virtual elements are identified using the same reference numbersas in FIG. 81A, and discussion of such physical and/or virtual elementswill not be repeated in the interest of brevity.

As illustrated in FIG. 62C, a user may gesture to bring up a thirdvirtual décor, different from the first and the second virtual decors.The third virtual décor may, for example, replicate a view of a beachscene and a different virtual picture. The gesture to bring up the thirdvirtual décor may be identical to the gesture to bring up the first andthe second virtual decors, the user essentially toggling, stepping orscrolling through a set of defined virtual decors for the physical roomor physical space (e.g., physical living room). Alternatively, eachvirtual décor may be associated with a respective gesture. Similarly,the user may enjoy a fourth virtual décor as well as shown in FIG. 62D(scene 6208)

As illustrated, a user interface tool may indicate that which of the setof virtual decors is currently selected and mapped to the physical roomor space.

FIG. 63 (scene 6300) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual room or virtualspace in the form of a virtual entertainment or media room, the userexecuting gestures to interact with a user interface virtual construct,according to one illustrated embodiment.

As illustrated in FIG. 63, the AR system may render a hierarchical menuuser interface virtual construct including a plurality of virtualtablets or touch pads, so at to appear in a user's field of view,preferably appearing to reside within a reach of the user. These allow auser to navigate a primary menu to access user defined virtual rooms orvirtual spaces, which are a feature of the primary navigation menu. Thevarious functions or purposes of the virtual rooms or virtual spaces maybe represented iconically. Based on the user's gestures, various iconsof the user interface may be moved or selected by the user. The ARsystem may retrieve data from the cloud server, similar to the processflow of FIG. 54, as needed.

FIG. 63 shows a user sitting in a physical living room space, and usingan AR system to experience a virtual room or virtual space in the formof a virtual entertainment or media room, the user executing gestures tointeract with a user interface virtual construct to provide input byproxy, according to one illustrated embodiment.

As illustrated in FIG. 63, the AR system may render a user interfacevirtual construct including a plurality of user selectable virtualelements, so at to appear in a user's field of view. The usermanipulates a totem to interact with the virtual elements of the userinterface virtual construct. The user, may for example, point a front ofthe totem at a desired one of the elements.

The user may also interact with the totem, for example tapping ortouching on a surface of the totem, indicating a selection of theelement at which the totem is pointing or aligned. The AR system detectsthe orientation of the totem and the user interactions with the totem,interpreting such as a selection of the element at which the totem ispointing or aligned. The AR system the executes a corresponding action,for example opening an application, opening a submenu, or rendering avirtual room or virtual space corresponding to the selected element.

The totem may replicate a remote control, for example remote controlscommonly associated with televisions and media players. In someimplementations, the totem may be an actual remote control for anelectronic device (e.g., television, media player, media streaming box),however the AR system may not actually received any wirelesscommunications signals from the remote control. The remote control mayeven not have batteries, yet still function as a totem since the ARsystem is relies on image that capture position, orientation andinteractions with the totem (e.g., remote control).

FIGS. 64A and 64B (scenes 6402 and 6404) show a user sitting in aphysical living room space, and using an AR system to experience avirtual room or virtual space in the form of a virtual entertainment ormedia room, the user executing gestures to interact with a userinterface virtual construct to provide input, according to oneillustrated embodiment.

As illustrated in FIG. 64A, the AR system may render a user interfacevirtual construct including an expandable menu icon that is alwaysavailable. The AR system may consistently render the expandable menuicon in a given location in the user's field of view, or preferably in aperipheral portion of the user's field of view, for example an upperright corner. Alternatively, AR system may consistently render theexpandable menu icon in a given location in the physical room orphysical space.

As illustrated in FIG. 64B, the user may gesture at or toward theexpandable menu icon to expand the expandable menu construct. Inresponse, the AR system may render the expanded expandable menuconstruct to appear in a field of view of the user. The expandable menuconstruct may expand to reveal one or more virtual rooms or virtualspaces available to the user. The AR system may consistently render theexpandable menu in a given location in the user's field of view, orpreferably in a peripheral portion of the user's field of view, forexample an upper right corner. Alternatively, AR system may consistentlyrender the expandable menu in a given location in the physical room orphysical space.

FIG. 65A (scene 6502) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual décor (i.e.,aesthetic skin or aesthetic treatment), the user executing pointinggestures to interact with a user interface virtual construct, accordingto one illustrated embodiment.

As illustrated in FIG. 65A, the AR system may render a user interfacetool which includes a number of pre-mapped menus. For instance, the ARsystem may render a number of poster-like virtual images correspondingto respective pieces of entertainment or media content (e.g., movies,sports events), from which the user can select via one or more pointinggestures. The AR system may render the poster-like virtual images to,for example, appear to the user as if hanging or glued to a physicalwall of the living room. Again, the AR system detects the mapcoordinates of the room, and displays the virtual posters in the rightsize and at the right orientation with respect to the mappedcoordinates, such that the posters appear to be placed on the wall ofthe room.

The AR system detects the user's gestures, for example pointing gestureswhich may include pointing a hand or arm toward one of the poster-likevirtual images. The AR system recognizes the pointing gesture orprojection based proxy input, as a user selection intended to triggerdelivery of the entertainment or media content which the poster-likevirtual image represents. The AR system may render an image of a cursor,with the cursor appearing to be projected toward a position in which theuser gestures. The AR system causes the cursor to tracking the directionof the user's gestures, providing visual feedback to the user, andthereby facilitating aiming to allow projection based proxy input.

FIG. 65B (scene 6504) shows a user sitting in a physical living roomspace, and using an AR system to experience a virtual décor (i.e.,aesthetic skin or aesthetic treatment), the user executing touchgestures to interact with a user interface virtual construct, accordingto one illustrated embodiment.

As illustrated in FIG. 65B, the AR system may render a user interfacetool which includes a number of pre-mapped menus. For instance, the ARsystem may render a number of poster-like virtual images correspondingto respective pieces of entertainment or media content (e.g., movies,sports events), from which the user can select via one or more touchgestures. The AR system may render the poster-like virtual images to,for example, appear to the user as if hanging or glued to a physicalwall of the living room.

The AR system detects the user's gestures, for example touch gestureswhich includes touches at least proximate an area in which one of theposter-like virtual images appears to be rendered. The AR systemrecognizes the touching gesture or virtual tablet or touch pad likeinteraction, as a user selection intended to trigger delivery of theentertainment or media content which the poster-like virtual imagerepresents.

FIG. 65C (6506) shows a user sitting in a physical living room space,and using an AR system to experience a piece of entertainment or mediacontent, the user executing touch gestures to interact with a userinterface virtual construct, according to one illustrated embodiment.

As illustrated in FIG. 65C, in response a user selection, the AR systemrenders a display of the selected entertainment or media content, and/orassociated virtual menus (e.g., high level virtual navigation menu, forinstance a navigation menu that allows selection of primary feature,episode, of extras materials). For example, the AR system may render adisplay of the selected entertainment or media content to the retina ofthe user's eyes, so that the selected entertainment or media contentappears in the field of view of the user as if displayed on a wall ofthe physical space. As illustrated in FIG. 65C, the display of theselected entertainment or media content may replace at least a portionof the first virtual décor.

As illustrated in FIG. 65C, in response the user selection, the ARsystem may also render a virtual tablet type user interface tool, whichprovides a more detailed virtual navigation menu than the high levelvirtual navigation menu. The more detailed virtual navigation menu mayinclude some or all of the menu options of the high level virtualnavigation menu, as well as additional options (e.g., retrieveadditional content, play interactive game associated with media title orfranchise, scene selection, character exploration, actor exploration,commentary). For instance, the AR system may render the detailed virtualnavigation menu to, for example, appear to the user as if sitting on atop surface of a table, within arm's reach of the user.

The AR system detects the user's gestures, for example touch gestureswhich includes touches at least proximate an area in which the moredetailed virtual navigation menu appears to be rendered. The AR systemrecognizes the touching gesture or virtual tablet or touch pad likeinteraction, as a user selection intended to effect delivery of theassociated entertainment or media content.

Referring now to FIGS. 66A-66J (scenes 6102-6120), another user scenariois illustrated. FIGS. 66A-66J illustrate an AR system implemented retailexperience, according to one illustrated embodiment.

As illustrated, a mother and daughter each wearing respective individualAR systems receive an augmented reality experience while shopping in aretail environment, for example a supermarket. As explained herein, theAR system may provide entertainment as well as facilitate the shoppingexperience. For example, the AR system may render virtual content, forinstance virtual characters which may appear to jump from a box orcarton, and/or offer virtual coupons for selected items. The AR systemmay render games, for example games based on locations throughout thestore and/or based on items on shopping list, list of favorites, or alist of promotional items. The augmented reality environment encourageschildren to play, while moving through each location at which a parentor accompanying adult needs to pick up an item. Even adults may play.

In another embodiment, the AR system may provide information about foodchoices, and may help users with their health/weight/lifestyle goals.The AR system may render the calorie count of various foods while theuser is consuming it, thus educating the user on his/her food choices.If the user is consuming unhealthy food, the AR system may warn the userabout the food so that the user is able to make an informed choice.

The AR system may subtly render virtually coupons, for example usingradio frequency identification (RFID) transponders and communications.For example, referring back to process flow of FIG. 55, the AR systemmay recognize the real world activity (shopping), and load informationfrom the knowledge database regarding shopping.

Based on recognizing the specific activity, the system may unlockmetadata, or display virtual content based on the recognized specificactivity. For example, the AR system may render visual affects tied orproximately associated with items, for instance causing a glowing affectaround box glows to indicate that there is metadata associated with theitem. The metadata may also include or link to a coupon for a discountor rebate on the item. The AR system may detect user gestures, and forexample unlocking metadata in response to defined gestures.

The AR system may recognize different gestures for different items. Forexample, as explained herein, a virtual animated creature may berendered so as to appear to pop out of a box holding a coupon for thepotential purchaser or customer. For example, the AR system may rendervirtual content that makes a user perceive a box opening. The AR systemallows advertising creation and/or delivery at the point of customer orconsumer decision.

The AR system may render virtual content which replicates a celebrityappearance. For example, the AR system may render a virtual appearanceof a celebrity chef at a supermarket, as will be described furtherbelow. The AR system may render virtual content which assists incross-selling of products. For example, one or more virtual affects maycause a bottle of wine to recommend a cheese that goes well with thewine. The AR system may render visual and/or aural affects which appearto be proximate the cheese, in order to attract a shopper's attention.The AR system may render one or more virtual affects in the field of theuser that cause the user to perceive the cheese recommending certaincrackers. The AR system may render friends who may provide opinions orcomments regarding the various produces (e.g., wine, cheese, crackers).

The AR system may render virtual affects within the user's field of viewwhich are related to a diet the user is following. For example, theaffects may include an image of a skinny version of the user, which isrendered in response to the user looking at a high calorie product. Thismay include an aural oral reminder regarding the diet. Similar to above(refer to process flow of FIG. 55), the AR system recognizes visualinput (here, a high calorie product) and automatically retrieves datacorresponding to the skinny version of the user to display. The systemalso uses map coordinates of the high calorie product to display theskinny version of the user right next to the physical product.

In particular, FIG. 66A shows mother with her daughter in tow, pushing ashopping cart from an entrance of a grocery store. The AR systemrecognizes the presence of a shopping cart or a hand on the shoppingcart, and determines a location of the user and/or shopping cart. Basedon such, the AR system automatically launches a set of relevantapplications, rendering respective user interfaces of the applicationsto the user's field of view. In other words, similar to the process flowof FIG. 55, the AR system recognizes the specific activity as shopping,and automatically retrieves data associated with the relevantapplications to be displayed in a floating user interface.

Applications may, for example, include a virtual grocery list. Thegrocery list may be organized by user defined criteria (e.g., dinnerrecipes). The virtual grocery list may be generated before the userleaves home, or may be generated at some later time, or even generatedon the fly, for example in cooperation with one of the otherapplications. The applications may, for example, include a virtualcoupon book, which includes virtual coupons redeemable for discounts orrebates on various products. The applications may, for example, includea virtual recipe book, which includes various recipes, table ofcontents, indexes, and ingredient lists.

Selection of a virtual recipe may cause the AR system to update thegrocery list. In some implementations, the AR system may update thegrocery list based on knowledge of the various ingredients the useralready has at home, whether in a refrigerator, freezer or cupboard. TheAR system may collect this information throughout the day as the userworks in the kitchen of their home. The applications may, for example,include a virtual recipe builder. The recipe builder may build recipesaround defined ingredients.

For example, the user may enter a type of fish (e.g., Atlantic salmon),and the recipe builder will generate a recipe that uses the ingredient.Selection of a virtual recipe generated by the recipe builder may causethe AR system to update the grocery list. In some implementations, theAR system may update the grocery list based on knowledge of the variousingredients the user already has at home. The applications may, forexample, include a virtual calculator, which may maintain a runningtotal of cost of all items in the shopping cart.

FIG. 66B shows mother with her daughter in a produce section. The motherweighs a physical food item on a scale. The AR system automaticallydetermines the total cost of the item (e.g., price per pound multipliedby weight) enters the amount into the running total cost. The AR systemautomatically updates the ‘smart’ virtual grocery list to reflect theitem. The AR system automatically updates the ‘smart’ virtual grocerylist based on location to draw attention to items on the grocery listthat are nearby the present location.

For example, the AR system may update the rendering of the virtualgrocery list to visually emphasize certain items (e.g., focused onfruits and vegetables in the produce section). Such may includehighlighting items on the list or moving close by items to a top of thelist. Further, the AR system may render visual effects in the field ofview of the user such that the visual affects appear to be around orproximate nearby physical items that appear on the virtual grocery list.

FIG. 66C shows the child selecting a virtual icon to launch a scavengerhunt application. The scavenger hunt application makes the child'sshopping experience more engaging and educational. The scavenger huntapplication may present a challenge, for example, involving locatingfood items from different countries around the world. Points are addedto the child's score as she identifies food items and puts them in hervirtual shopping cart. Based on the input received by the child, the ARsystem may retrieve data related to the scavenger hunt from the cloud(e.g., instructions for the scavenger hunt, etc.) and transmit back tothe user device so that the scavenger hunt instructions are timelydisplayed to the child.

FIG. 66D shows the child finding and gesturing toward a bonus virtualicon, in the form of a friendly monster or an avatar. The AR system mayrender unexpected or bonuses virtual content to the field of view of thechild to provide a more entertaining and engaging user experience. TheAR system, detects and recognizes the gesture of pointing toward themonster, and unlocks the metadata associated with the friendly monsteror avatar. By gesturing toward the monster, the AR system recognizes themap coordinates of the monster, and therefore unlocks it based on theuser's gesture. The bonus information is then retrieved from the cloudand displayed in the appropriate map coordinates next to the friendlymonster, for instance.

FIG. 66E show the mother and daughter in a cereal aisle. The motherselects a particular cereal to explore additional information, forexample via a virtual presentation of metadata. The metadata may, forexample, include: dietary restrictions, nutritional information (e.g.,health stars), product reviews and/or product comparisons, or customercomments. Rendering the metadata virtually allow the metadata to bepresented in a way that is easily readable, particular for adults howmay have trouble reading small type or fonts. Similar to the processflow of FIG. 55, the system may recognize the real-worldactivity/real-world object, retrieve data associated with it, andappropriately display the virtual information associated with theparticular cereal.

As also illustrated in FIG. 66E, an animated character (e.g., ToucanSam®) is rendered and may be presented to the customers with any virtualcoupons that are available for a particular item. The AR system mayrender coupons for a given product to all passing customers, or only tocustomers who stop. Alternatively or additionally, the AR system mayrender coupons for a given product to customers who have the givenproduct on their virtual grocery list, or only to those who have acompeting product on their virtual grocery list. Alternatively oradditionally, the AR system may render coupons for a given product basedon knowledge of a customer's past or current buying habits and/orcontents of the shopping cart. Here, similar to FIG. 55, the AR systemmay recognize the real-world activity, load the knowledge baseassociated with the virtual coupons, and based on the user's specificinterest or specific activity, may display the relevant virtual couponsto the user.

As illustrated in FIG. 66F, the AR system may render an animatedcharacter (e.g., friendly monster) in the field of view of at least thechild. The AR system may render the animated character so as to appearto be climbing out of a box (e.g., cereal box). The sudden appearance ofthe animated character may prompt the child to start a game (e.g.,Monster Battle). The child can animate or bring the character to lifewith a gesture. For example, a flick of the wrist may cause the ARsystem to render the animated character bursting through the cerealboxes.

FIG. 66G shows the mother at an end of an aisle, watching a virtualcelebrity chef (e.g., Mario Batali) presentation via the AR system. Thecelebrity chef may demonstrate a simple recipe to customers. Allingredients used in the demonstrated recipe may be available at the endcap. This user scenario may utilize the process flow of FIGS. 53 and 54.The AR system essentially allows the celebrity chef to pass over a pieceof his world to multiple users. Here, based on detecting a location atthe store, the AR system retrieves data from the passable worldassociated with the celebrity chef's live performance, and sends backthe relevant information to the user's device.

In some instances, the AR system may present the presentation live. Thismay permit questions to be asked of the celebrity chef by customers atvarious retail locations. In other instances, the AR system may presenta previously recorded presentation.

The AR system may capture the celebrity chef presentation via, forexample, a 4D light field. The presentation may likewise be presentedvia a 4D light field provided to the retina of the user's eyes. Thisprovides a realistic sense of depth, and the ability to circle to thesides and perceive the celebrity as if actually present in the retailenvironment.

In some implementations, the AR system may capture images of thecustomers, for example via inward facing cameras carried by eachcustomer's individual head worn component. The AR system may provide acomposited virtual image to the celebrity of a crowd composed of thevarious customers.

FIG. 66H shows the mother in a wine section of the grocery store. Themother may search for a specific wine using a virtual user interface ofan application. The application may be a wine specific application, anelectronic book, or a more general Web browser. In response to selectionof a wine, the AR system may render a virtual map in the field of viewof the user, with directions for navigating to the desired wine (similarto the process flow of FIG. 47).

The AR, based on user input, identifies the user interface desired bythe user, retrieves data associated with the user interface, anddisplays the user interface along the right map coordinates in thephysical space of the user. Here, for example, the location at which theuser interface is rendered may be tied to the map coordinates of theshopping cart. Thus, when the shopping cart moves, the user interfacemoves along with the shopping cart as well.

While the mother is walking through the aisles, the AR system maycapture may render data, which appear to be attached or at leastproximate respective bottles of wines to which the data relates. Thedata may, for example, include recommendations from friends, wines thatappear on a customer's personal wine list, and/or recommendations fromexperts. The data may additionally or alternatively include food paringsfor the particular wine.

FIG. 66I shows the mother and child concludes their shopping experience.The mother and child may, for example, by walking onto, across orthrough a threshold. The threshold may be implemented in any of a largevariety of fashions, for example as a suitably marked map. The AR systemdetects passage over or through the threshold, and in response totals upthe cost of all the groceries in the shopping cart. The AR system mayalso provide a notification or reminder to the user, identifying anyitems on the virtual grocery list where are not in the shopping cart andthus may have been forgotten. The customer may complete the check-outthrough a virtual display,—no credit card necessary.

As illustrated in FIG. 66J, at the end of the shopping experience, thechild receives a summary of her scavenger hunt gaming experience, forexample including her previous high score. The AR system may render thesummary as virtual content, at least in the field of view of the child.

FIG. 67 (scene 6700) shows a customer employing an AR system in a retailenvironment, for example a bookstore, according to one illustratedembodiment. The customer opens up a book totem. The AR system detectsthe opening of the book totem, and in response renders an immersivevirtual bookstore experience in the user's field of view. The virtualbookstore experience may, for example, include reviews of books,suggestions, and author comments, presentations or readings. The ARsystem may render additional content, for example virtual coupons.

The virtual environment combines the convenience of an online bookstorewith the experience of a physical environment.

User Experience Health Care Example

FIGS. 68A-68F (scenes 6802-6812) illustrate use of an AR system in ahealth care related application or physical environment, which mayinclude recovery and/or rehabilitation, according to one illustratedembodiment.

In particular, FIG. 68A shows a surgeon and surgical team, including avirtually rendered consulting or visiting surgeon, conducting apre-operative planning session for an upcoming mitral valve replacementprocedure. Each of the health care providers is wearing a respectiveindividual AR system.

As noted above, the AR system renders a visual representation of theconsulting or visiting surgeon. As discussed herein, the visualrepresentation may take many forms, from a very simple representation toa very realistic representation.

The AR system renders a patient's pre-mapped anatomy (e.g., heart) in 3Dfor the team to analyze during the planning. The AR system may renderthe anatomy using a light field, which allows viewing from any angle ororientation. For example, the surgeon could walk around the heart to seea back side thereof.

The AR system may also render patient information. For instance, the ARsystem may render some patient information (e.g., identificationinformation) so as to appear on a surface of a physical table. Also forinstance, the AR system may render other patient information (e.g.,medical images, vital signs, charts) so as to appear on a surface of oneor more physical walls. Similar to the process flow of FIG. 55, the ARsystem may detect and recognize input (e.g, here the users mayexplicitly request to see virtual representation of the pre-mappedanatomy of the heart). Here, based on input, the AR system may retrievethe data from the cloud server, and transmit it back to the user'sdevices. The system also uses the map coordinates of the room to displaythe virtual content in the center of the room so that it can be viewedby multiple users sitting around the table.

As illustrated in FIG. 68B, the surgeon is able to reference thepre-mapped 3D anatomy (e.g., heart) during the procedure. Being able toreference the anatomy in real time, may for example, improve placementaccuracy of a valve repair. Outward pointed cameras capture imageinformation from the procedure, allowing a medical student to observevirtually via the AR system from her remote classroom. The AR systemmakes a patient's information readily available, for example to confirmthe pathology, and avoid any critical errors.

FIG. 68C shows a post-operative meeting or debriefing between thesurgeon and patient. During the post-operative meeting, the surgeon isable to describe how the surgery went using a cross section of virtualanatomy or virtual 3D anatomical model of the patient's actual anatomy.The AR system allows the patient's spouse to join the meeting virtuallywhile at work. Again, the AR system may render a light field whichallows the surgeon, patient and spouse to inspect the virtual 3Danatomical model of the patient's actual anatomy from a desired angle ororientation.

FIG. 68D shows the patient convalescing in a hospital room. The ARsystem allows the patient to perceive any type of relaxing environmentthat the patient may desire, for example a tranquil beach setting. Here,similar to process flow of FIG. 54, the AR system retrieves dataassociated with the beach setting from the cloud, maps the roomcoordinates in order to display the beach setting virtual décor alongthe desired wall of the hospital room.

As illustrated in FIG. 68E, the patient may practice yoga or participatein some other rehabilitation during the hospital stay and/or afterdischarge. The AR system allows the patient to perceive a friendvirtually rendered in a virtual yoga class. Similar to process flow ofFIG. 53, multiple users are able to pass a piece of their passable worldto each other.

More specifically, the AR system updates the passable world model basedon the changes to the each of the user's position, location, and imagedata, as seen by their FOV cameras and other image sources, determinesthe 3D points based on the images captured by the FOV cameras, andrecognizes various objects (and attaches semantic information). Here,information regarding the physical space is continually updated in thepassable world model which is transmitted to the other users that arenot physically present in the room where the first user is doing yoga.Similarly, information about the other user's movements etc. are alsoupdated on the passable world model, which is transmitted to the firstuser such that the user views the avatars of the user in the samephysical room.

As illustrated in FIG. 68F, the patient may participate inrehabilitation, for example riding on a stationary bicycle during thehospital stay and/or after discharge. The AR system renders, in theuser's field of view, information about the simulated cycling route(e.g., map, altitude, and distance), patient's performance statistics(e.g., power, speed, heart rate, ride time). The AR system render avirtual biking experience, for example including an outdoor scene,replicating a ride course such as a favorite physical route.

Additionally or alternatively, the AR system renders a virtual avatar asa motivational tool. The virtual avatar may, for example, replicate aprevious ride, allowing the patient to compete with their own personalbest time. Here, similar to the process flow of FIG. 55, the AR systemdetects the user's real-world activity (cycling) and loads a knowledgebased related to cycling. Based on the user's specific activity (e.g.,speed of cycling, etc.), the AR system may retrieve relevant information(e.g, statistics, motivational tools, etc.) and display the informationto the user at the appropriate location by mapping the coordinates ofthe physical space at which the user is cycling.

User Experience Work/Manual Labor Example

FIG. 69 (scene 6900) shows a worker employing an AR system in a workenvironment, according to one illustrated embodiment.

In particular, FIG. 69 shows a landscaping worker operating machinery(e.g., lawn mower). Like many repetitive jobs, cutting grass can betedious. Workers may lose interest after some period of time, increasingthe probability of an accident. Further, it may be difficult to attractqualified workers, or to ensure that workers are performing adequately.

The worker wears an individual AR system, which renders virtual contentin the user's field of view to enhance job performance. For example, theAR system may render a virtual game, where the goal is to follow avirtually mapped pattern. Points are received for accurately followingthe pattern and hitting certain score multipliers before they disappear.Points may be deducted for straying from the pattern or straying tooclose to certain physical objects (e.g., trees, sprinkler heads,roadway).

While only one example environment is illustrated, this approach can beimplemented in a large variety of work situations and environments. Forexample, a similar approach can be used in warehouses for retrievingitems, or in retail environments for stacking shelves, or for sortingitems such as mail. This approach may reduce or eliminate the need fortraining, since a game or pattern may be provided for many particulartasks.

Any of the devices/servers in the above-described systems may include abus or other communication mechanism for communicating information,which interconnects subsystems and devices, such as processor, systemmemory (e.g., RAM), static storage device (e.g., ROM), disk drive (e.g.,magnetic or optical), communication interface (e.g., modem or Ethernetcard), display (e.g., CRT or LCD), input device (e.g., keyboard,touchscreen). The system component performs specific operations by theprocessor executing one or more sequences of one or more instructionscontained in system memory.

Such instructions may be read into system memory from another computerreadable/usable medium, such as static storage device or disk drive. Inalternative embodiments, hard-wired circuitry may be used in place of orin combination with software instructions to implement the invention.Thus, embodiments of the invention are not limited to any specificcombination of hardware circuitry and/or software. In one embodiment,the term “logic” shall mean any combination of software or hardware thatis used to implement all or part of the invention.

The term “computer readable medium” or “computer usable medium” as usedherein refers to any medium that participates in providing instructionsto processor 1407 for execution. Such a medium may take many forms,including but not limited to, non-volatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks,such as disk drive. Volatile media includes dynamic memory, such assystem memory. Common forms of computer readable media includes, forexample, floppy disk, flexible disk, hard disk, magnetic tape, any othermagnetic medium, CD-ROM, any other optical medium, punch cards, papertape, any other physical medium with patterns of holes, RAM, PROM,EPROM, FLASH-EPROM, any other memory chip or cartridge, or any othermedium from which a computer can read.

In an embodiment of the invention, execution of the sequences ofinstructions to practice the invention is performed by a singlecomputing system. According to other embodiments of the invention, twoor more computing systems coupled by a communication link (e.g., LAN,PTSN, or wireless network) may perform the sequence of instructionsrequired to practice the invention in coordination with one another. Thesystem component may transmit and receive messages, data, andinstructions, including program, i.e., application code, throughcommunication link and communication interface. Received program codemay be executed by the processor as it is received, and/or stored indisk drive, or other non-volatile storage for later execution.

Various exemplary embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices. The methods may comprise the act of providing such a suitabledevice. Such provision may be performed by the end user. In other words,the “providing” act merely requires the end user obtain, access,approach, position, set-up, activate, power-up or otherwise act toprovide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Exemplary aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.In other words, use of the articles allow for “at least one” of thesubject item in the description above as well as claims associated withthis disclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element—irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

What is claimed is:
 1. A method of retrieving and presenting data, themethod comprising: receiving a visual user input in the form of agesture performed by a user; based at least partly on interpreting thegesture, sending a request to a cloud server in response to theinterpreted gesture; receiving, from the cloud server, the data from aknowledge base connected to the cloud server, the data retrieved fromthe knowledge base by the cloud server in response to the request, thedata including a navigation menu; receiving, from the cloud server, aset of map points associated with a position of a hand of the user;displaying a virtual user interface comprising the navigation menu usingthe set of map points, wherein the virtual user interface is displayedin real time at a position that corresponds to a current position of thehand as described by the set of map points; detecting a further gesture,the further gesture triggering an expansion of the plurality of firstlevel menu items by spreading apart fingers of a hand, wherein theexpansion of the plurality of first level menu items renders a pluralityof second level menu items that appears on the finger of the user;recognizing, based on the detected further gesture, a second activecommand to create a second virtual user interface comprising theplurality of second level menu items such that the second virtual userinterface, when viewed by the user, appears to be stationary at a secondposition of the hand; and rendering, in real-time and based on thedetected further gesture, the second virtual user interface at thedetermined map points associated with the second position of the handsuch that the plurality of first level menu items of the virtual userinterface and the plurality of second level menu items of the secondvirtual user interface are simultaneously viewable on the finger of theuser, the second level menu items being derived from the first levelmenu items as lower level menu items.
 2. The method of claim 1, furthercomprising determining if the user input is valid before sending therequest to the cloud server.
 3. The method of claim 1, furthercomprising selecting the knowledge base from a plurality of knowledgebases connected to the cloud server in response to the request sent tothe cloud server.
 4. The method of claim 1, wherein differentpredetermined gestures are respectively mapped to differentcorresponding commands, and wherein interpreting the meaning of thegesture comprises matching the gesture to one of the predeterminedgestures, thereby determining the command corresponding to the onepredetermined gesture.
 5. The method of claim 1, wherein the gesture isa finger/hand gesture.
 6. The method of claim 5, wherein the finger/handgesture is one of an opening or holding up of the palm of the hand. 7.The method of claim 1, wherein additional user input is received via thevirtual user interface.
 8. The method of claim 1, wherein a camerareceives the visual user input from the user.
 9. The method of claim 8,wherein the camera is an FOV camera.
 10. The method of claim 1, whereinan item, icon, or field of the navigation menu is displayed on each of aplurality of fingers of the user, and wherein an item, icon, or field ofthe navigation menu is not displayed on a thumb of the user.
 11. Themethod of claim 10, further comprising: receiving another visual userinput from the user in the form of another gesture comprising touchingone of the plurality of fingers with the thumb; interpreting the meaningof the other gesture; sending another request to the cloud server; andselecting the item, icon, or field of the navigation menu that isdisplayed on the touched finger in response to the request.
 12. Themethod of claim 11, wherein the item, icon, or field of the navigationmenu comprises one of an instance application, function, menu, tool,model, game, and virtual room or virtual space.
 13. The method of claim10, further comprising: receiving another visual user input from theuser in the form of another gesture comprising making a circle motion ina palm of the hand of the user with a finger of the other hand of theuser; interpreting the meaning of the other gesture; sending anotherrequest to the cloud server; and scrolling the displayed items, icons,or fields of the navigation menu successively between the plurality offingers.
 14. The method of claim 13, wherein if the circle motion isclockwise, the displayed items, icons, or fields of the navigation aresuccessively scrolled in a first direction between the plurality offingers, and if the circle motion is counterclockwise, the displayeditems, icons, or fields of the navigation are successively scrolled in asecond opposite direction between the plurality of fingers.
 15. Themethod of claim 1, wherein the cloud server comprises clusters ofcomputer systems located different geographically dispersed locations.16. The method of claim 1, wherein the cloud server comprises geometrymapping data, semantic mapping data, and user personal information.